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CHAPTER1 OVERVIEW 


This technical report provides an overview of the New York State Alternate Assessment 
(NYSAA), including a description of the purpose of the NYSAA, the processes utilized to develop and 
implement the NYSAA program, and Stakeholder involvement in those processes. By comparing the 
intent of the NYSAA with its process and design, the validity of the assessment’s use can be evaluated. 
The 2013-14 NYSAA incorporates a new test design for all content areas. For the 2013-14 NYSAA, 
English Language Arts (ELA) and mathematics Extensions, which are aligned to the Common Core 
Learning Standards (CCLS), were developed. For science and social studies Alternate Grade-Level 
Indicators (AGLIs) aligned to the New York State learning standards, development occurred in 2006—07 
and 2007-08. The processes for developing the Extensions and AGLIs are presented in detail. 
Stakeholder input in the development of the overall NYSAA process itself is described in detail, 
including the content alignment of the Extension and AGLI design following the new Blueprint/test 
design, the Assessment Task development, the teacher trainings for administration, the scoring 


trainings and process, and the standard setting. 


1.1 PURPOSE OF THE REPORT 


The purpose of this report is to document the technical aspects of the 2013-14 NYSAA. During 
the 2013-14 school year, approximately 21,737 students in Grades 3 through 8 and in high school 
participated in the administration of the NYSAA. ELA and mathematics were assessed at the Grades 3 
through 8 and high school levels; science was assessed at the Grades 4, 8, and high school levels; and 
social studies was assessed at the high school level. 

Several technical aspects of the NYSAA are described in an effort to contribute to evidence 
supporting the validity of NYSAA score interpretations. Because the interpretations of the test scores 
are evaluated for validity, not the test itself, this report presents documentation to substantiate intended 
interpretations (AERA, APA, & NCME, 2014). Each chapter in this section contributes important 
information to the validity argument by addressing one or more of the following aspects of the NYSAA: 
AGLI and Assessment Task development, alignment, administration, scoring, reliability, standard 
setting, and achievement levels. 

The Standards for Educational and Psychological Testing (AERA, APA, & NCME, 2014) 
provides a framework for describing sources of evidence that should be considered when constructing 
an argument for assessment validity. These evidence sources include those in five general areas: test 
content, response processes, internal structure, relationship to other variables, and consequences of 


testing. Although each of these sources may speak to a different aspect of validity, they are not distinct 
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types of validity. Instead, each contributes to a body of evidence about the comprehensive validity of 


score interpretations. 


1.2 ORGANIZATION OF THIS REPORT 


This report is organized based on the conceptual flow of the NYSAA as a multi-year long 
process, which includes Blueprint design/development (completed in 2012-13), Extension (completed 
in 2012-13) and AGLI (completed in 2006—07 and 2007-08) development, Assessment Task 
development (completed in 2012-13), administration (completed in 2013-14), scoring (completed in 
2013-14), standard setting (completed in 2013-14), technical characteristics, and validity. The 


appendices contain supporting documentation. 


1.3 CURRENT YEAR UPDATES 


The NYSAA was redesigned beginning with the 2013-14 administration. The overall structure of 
the NYSAA remains consistent with past practice in that it is a datafolio-style assessment that includes 
student performance data and evidence, and is designed to assess students with severe cognitive 
disabilities. The redesign is comprised of two major changes. The first is the implementation of a new 
test design. This test design is described in the NYSAA Administration Manual (September 2013). The 
second major part of this redesign is the implementation of new test blueprints and the replacement of 
AGLIs with a set of Extensions aligned to the CCLS for ELA and mathematics. With some 
modifications, the AGLIs continued to be used to assess science and social studies, as appropriate per 
grade assessed. 

The Department worked closely with special education and general education teacher 
committees from across the State and with the contractor team from Measured Progress in developing 
the new assessment. Teacher committees worked to develop Essences for the CCLS in ELA and 
mathematics, and then to develop Extensions aligned to those Essences. For each Extension, the 
committees developed Assessment Tasks. For the ELA and mathematics content areas, five standards 
have been identified that provide a broad representation of the content. Each student’s datafolio will 
include performance and evidence of their knowledge, skills, and understanding for each of the five 
standards. Two dates of student performance will be documented for each standard in all content 
areas. Early in the administration period, teachers should administer a baseline data point to collect 
performance data and evidence that confirm the student has not yet mastered the assessed skill. 
Based on the outcome of the baseline data point, it may be necessary to adjust the level of complexity 
(choose another task at a higher or lower level of complexity). Between November and January, 
teachers should provide instruction and evaluation to gauge student growth. Before the end of the 


administration period, a final administration will be collected and documented in the datafolio. 
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The Extensions and AGLIs, and the Assessment Tasks aligned to each Extension or AGLI, are 
structured by Level of Complexity. A teacher may choose to assess a student at the less complex, 
middle, or more complex level, based on the baseline administration data. The assessed Level of 
Complexity may be taken into account when determining the student’s performance level for a content 


area. 
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CHAPTER2 THE STATE ASSESSMENT SYSTEM 


In New York State, both the general large-scale assessments and the alternate assessment test 
students on English Language Arts (ELA) and mathematics curriculum content taught during Grades 3 
through 8 and during high school; on science content taught during Grades 4, 8, and during high 
school; and on social studies content taught during high school. All students participate in the statewide 
assessment program through the following: the general assessments with or without accommodations, 
the alternate assessment with or without accommodations, or a combination of the general and 


alternate assessments with or without accommodations. 


2.1 INTRODUCTION 


The New York State Alternate Assessment (NYSAA) is designed to provide a snapshot in time 
of an individual student’s performance. A broader picture will emerge as the student results on the 
NYSAA are reviewed, along with results on other classroom and district assessments. 

The NYSAA is a datafolio-style assessment that measures how well students with severe 
cognitive disabilities meet the standards at alternate achievement levels. All students, including those 
with severe cognitive disabilities, are required by federal law to have access to the general education 
curriculum. The New York State Education Department (the Department) has aligned Extensions with 
Common Core Learning Standards (CCLS) in ELA and mathematics, and Alternate Grade-Level 
Indicators (AGLIs) with the core curriculum in science and social studies, for the administration of the 
NYSAA. The content-area subject matter assessed by the NYSAA is clearly related to the grade-level 
content. While the content is reduced in scope and complexity, students with severe cognitive 
disabilities are held to high expectations in order to achieve the standards. The Extensions and AGLIs 
afford students a richer learning experience. 

School districts across the United States are required to assess all students according to federal 
statute and State regulations. Assessment results tell educators how students are progressing and 
signal where changes may need to be made in curriculum and/or instruction at the district, school, and 
classroom levels. Teachers should assess students in all areas (academic, social, etc.) on an ongoing 
basis, as part of the instruction cycle. 

The No Child Left Behind (NCLB) Act of 2001 and the NYSAA are, in part, designed to raise 
expectations for students’ academic achievement. Students with severe cognitive disabilities, when 
given the appropriate instruction and access to the general education curriculum, have demonstrated 
progress in their knowledge, skills, and understanding in academic content areas that were not initially 


anticipated by school personnel or parents. Higher expectations require that students with severe 
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cognitive disabilities have access to the general education curriculum and be provided with specialized 


instruction, as well as participate in national, state, and local assessment programs. 
The administration period for the 2013-14 NYSAA was September 29, 2013, through February 
7, 2014. The scoring period for the 2013—14 NYSAA was March 10, 2014, to May 1, 2014. The general 


sequence of events for administering the NYSAA is highlighted below. 


Summary of NYSAA Events 


id, 


Each student’s Committee on Special Education (CSE) determines how a student 
participates in the New York State Testing Program. The CSE uses the Department’s 
guidelines regarding eligibility and participation criteria to guide their decision-making. 


For each content area assessed, the student's instructional team, headed by the Lead 
Special Education Teacher (teacher), considers the most appropriate level of complexity 
for the student in each content area assessed. Five Extensions are required in ELA and 
mathematics, and two AGLIs are required for science and social studies. 


Parents meet with the teacher to discuss how the NYSAA is administered and which 
specific AGLIs will be used to assess their child. 


Members of the student's instructional team conduct the baseline data point early in the 
administration period and document and rate student performance. Based on the results 
of the baseline assessment, the teacher will determine whether it is necessary to select 
another. The baseline data point serves two purposes: first, to confirm that the 
appropriate Level of Complexity has been selected and, second, to confirm that the 
student has not already mastered the selected skill. The baseline score cannot be higher 
than 74%. 


Once the baseline administration confirms the task to be assessed, the instructional 
team provides instruction on the assessed skill, continuing to evaluate student progress 
until it appears the student has acquired the skill. 


Following the instructional period, a final data point is administered and scored for Level 
of Accuracy. The date of the final data point should not be less than 15 school days after 
the date of the baseline data point, and should occur as close to the end of the 
administration period as possible (no later than February 7, 2014). Similar items and 
materials should be used for both the baseline and final administrations. 


The teacher assembles a datafolio containing the evidence of student performance and 
the percentages of the student’s Level of Accuracy. The completed datafolio is submitted 
to the building administrator on or before the last day of the administration period for 
shipping to the regional Scoring Institute. 


The NYSAA datafolios are scored at regional NYSAA Scoring Institutes during the 
scoring period defined by the Department. 


Student reports are created and are made available to school districts, teachers, and 
parents. 


2013-14 NYSAA Technical Report: Chapter 2—The State Assessment System -5- 


2.2 ALTERNATE ASSESSMENT BASED UPON ALTERNATE ACHIEVEMENT 
STANDARDS 


Up to 1% of New York State students in the grades tested may show academic proficiency 
through administration of an alternate assessment based on alternate achievement standards. The 
NYSAA is designed for those students with such severe cognitive disabilities that they are unable, even 
with the best instruction and appropriate accommodations, to participate in a general New York State 
assessment. The NYSAA is designed under the guiding philosophy that alternate achievement 
standards are built upon measurable, targeted skills linked to the New York State Grade-Level 
Indicators in ELA, mathematics, science, and social studies. However, the alternate achievement 
standards represent student performance at lower levels of breadth, depth, and complexity than those 


found in the general assessments. 


2.3 THE ALTERNATE ASSESSMENT SYSTEM 


The Individuals with Disabilities Education Act of 1997 (IDEA of 1997) requires that students 
with disabilities be included in each state’s system of accountability and have access to the general 
curriculum. The federal reauthorization of the Elementary and Secondary Education Act, known as the 
No Child Left Behind (NCLB) Act of 2001, also speaks to the inclusion of all children in a state’s 
accountability system by requiring states to report achievement for all students, as well as for groups of 
students on a disaggregated basis. These federal laws reflect an ongoing concern about equity: All 
students need to be academically challenged and taught to high standards. It is also necessary that all 
students be involved in the educational accountability system. Alternate achievement standards are 
reduced in breadth, depth, and complexity, but maintain linkage to the same general curriculum 
standards taught to all students. 

The IDEA of 1997 and the NCLB Act of 2001 clearly outline that all students, regardless of 
disability, participate in a statewide assessment system and be held accountable to the state standards. 
The NYSAA was developed to meet the requirements of these federal mandates; to provide a 
technically sound method to observe and record student achievement; to represent the breadth and 
depth of statewide content; to promote access to the general curriculum; to provide critical information 
to the CSE for use in the development of Individualized Education Programs (IEPs); and to meet 
criteria for alignment, access, burden, bias, sensitivity, and age appropriateness for students with 
severe cognitive disabilities. The 2013-14 NYSAA was the first year of implementation of the new test 


design. 
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2.4 PURPOSE OF THE ALTERNATE ASSESSMENT SYSTEM 


The NYSAA measures the achievements of students with severe cognitive disabilities relative to 
the New York State learning standards, using alternate achievement levels based on a datafolio 
approach (as described in the next section). To ensure that this student population has access to the 
general education curriculum, for the NYSAA administration, the Department aligned the Extensions 
and AGLIs (discussed in the next section) with the CCLS in ELA and mathematics and the core 
curriculum’s Grade-Level Expectations in science and social studies. 

The NYSAA is, in part, designed to raise expectations for students’ academic achievement. 
Experience has shown that students with severe cognitive disabilities, when given appropriate 
instruction and access to the general education curriculum, demonstrate unanticipated progress in their 
knowledge, skills, and understanding in academic content areas. Prior to 2006-07, access to the 
general education curriculum was not necessarily a part of instructional programs for students with 
severe cognitive disabilities. In a recent survey of teachers who administered the NYSAA in 2013-14, 
40% agreed that the Extensions and AGLIs assessed in the NYSAA made the grade-level core 
curricula more accessible, and said, the AGLIs are used in planning daily instruction. 

The process for assessing the academic achievements of students who have severe cognitive 
disabilities and who are eligible for the NYSAA is outlined through structured guidelines and steps in 
the 2013-14 NYSAA Administration Manual (accessible 
at http://www.p12.nysed.gov/assessment/nysaa/archive-13.html). The process for datafolio 
development (see Chapter 7) supports the procedural validity for assessing students with severe 
cognitive disabilities while being flexible enough to meet each individual student’s learning needs and 


modalities. 


2.5 TEST USE AND DECISIONS BASED ON ALTERNATE ASSESSMENT 


New York State conducts a statewide assessment program on an annual basis for all students 
in Grades 3 through 8 and in high school. The NYSAA ensures that students with severe cognitive 
disabilities are included in the New York State Testing Program and that their results are included in all 
Adequate Yearly Progress (AYP) determinations. 

Assessment based on Extensions and AGLIs is accomplished via datafolios. A datafolio is a 
collection of evidence of a student’s academic performance that is compiled by the student’s 
instructional team and scored by qualified Scorers. By gathering performance data, the instructional 
team can provide parents/families/guardians and the CSE with an understanding of the student’s 
knowledge, skills, and understanding as they relate to the CCLS in ELA and mathematics, and the New 
York State learning standards in science and social studies. The CSE can use the datafolio to 


understand the student’s achievement relative to these standards and to contribute to the development 
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of the student’s IEP. Datafolios are scored during a standardized scoring period each spring. The 
NYSAA student reports are generally available in the fall following administration. 

Performance levels, based on alternate academic achievement standards, were developed 
through a rigorous standard-setting process in June 2014. Alternate Performance Level Descriptors 
(APLDs) that outline the knowledge, skills, and understanding that a student may demonstrate within 
each grade and content area were edited and refined by panelists during the standard-setting process. 
APLDs, along with datafolios, provide information to parents/families/guardians, the CSE, and the 
instructional team regarding potential modifications or adjustments to the student’s instructional 


program. 


2.6 BACKGROUND AND GENERAL FORMAT 


A datafolio is a collection of evidence of a student’s academic performance compiled by the 
student's instructional team and scored by qualified Scorers. Instructional team members document 
student performance by documenting the student’s Level of Accuracy percentage as he or she 
performs an Assessment Task on two different dates, a baseline data point and a final data point, within 
the administration period. To verify this documentation, each datafolio must include student work 
products, Data Collection Sheets, photographs, or digital video and/or audio recordings. Teachers 
complete the required forms and submit all documentation and evidence in a binder or fastened folder 
for regional scoring. 

Teachers are provided with a NYSAA Administration Manual that outlines all of the assessment 
requirements, steps for compiling a datafolio, forms, and the NYSAA Frameworks as an appendix. The 
NYSAA Frameworks include an introduction, and the NYSAA Test Blueprints outline the standard that 
will be assessed via the alternate assessment for each grade. The Test Blueprints illustrate, for each 
content area (i.e., ELA, mathematics, science, and social studies), the major areas of the standards 
focus that teachers must assess at each grade. For both ELA and mathematics, five standards are 
assessed for each student and, in science and social studies, two standards are assessed. In ELA and 
mathematics, teachers select an Extension from one of three Levels of Complexity, based on their 
students’ needs. For students taking the NYSAA in science (grades 4, 8, and high school) and social 
studies (high school), teachers select an AGLI from one of three Levels of Complexity. 

Teachers must identify one Extension or AGLI based on the student’s assessed grade level. For 
each Extension or AGLI, the teacher must collect and document student performance data from an 
Assessment Task administered on three separate dates. One piece of verifying evidence must be 
submitted to demonstrate the student performance for each of the documented data points (baseline 


and final). 
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2.7 TESTING ACCOMMODATIONS 


The CSE determines whether a student will participate in the alternate assessment with or 


without accommodations. Guidelines regarding accommodations are provided in the NYSAA 


Administration Manual. The CSE determines which testing accommodations are required, based on the 


student’s documented needs. Testing accommodations: 


are consistent with the student’s IEP; 


are designed to allow the student to demonstrate his or her knowledge, skills, and 
understanding with greater independence; 


do not change the level of the assessment, the construct of the assessment, or the 
criteria of the Assessment Task; and 


are provided to the student during instruction and not just for assessment. 


For more information on testing accommodations, refer to Test Access and Accommodations for 


Students with Disabilities: Policy and Tools to Guide Decision-Making and Implementation (May 2006) 


at www.p12.nysed.gov/specialed/publications/policy/testaccess/policyquide.htm. 
Frequently asked questions about testing accommodations and the NYSAA can be found 


at www.p12.nysed.gov/assessment/nysaa/home.html. 
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CHAPTER3S THE STUDENTS 


New York State conducts a statewide testing program on an annual basis for all students in 
Grades 3 through 8 and in high school. The New York State Alternate Assessment (NYSAA) is a part of 
this statewide testing program. Designed for students with severe cognitive disabilities, the NYSAA 
measures student progress toward meeting the learning standards established for all students in the 
academic content areas of English Language Arts (ELA), mathematics, science, and social studies. 
The NYSAA ensures that students with severe cognitive disabilities are included in the State 
Assessment Program and that their results are accounted for as required by the No Child Left Behind 
(NCLB) Act of 2001 and the Individuals with Disabilities Education Act (IDEA) of 1997. 


3.1 TARGET POPULATION 


The target population for the NYSAA is extremely specific, and participation is limited to 
students with severe cognitive disabilities. The eligibility and participation criteria provide a definition of 
a student with a severe disability in accordance with section 100.1 of the Regulations of the 
Commissioner of Education. For reference, this information is provided in the NYSAA Administration 
Manual and on the Web site of the New York State Education Department (the Department). 

“Students with severe disabilities” refers to students who have limited cognitive abilities, 
combined with behavioral and/or physical limitations, and who require highly specialized educational 
and/or social, psychological, and medical services in order to maximize their full potential for useful and 
meaningful participation in society, and for self-fulfillment. Students with severe disabilities may 
experience severe speech, language, and/or perceptual-cognitive impairments and challenging 
behaviors that interfere with learning and socialization opportunities. These students may also have 
extremely fragile physiological conditions and may require personal care, physical/verbal supports, and 
assistive technology devices. 

The process of determining eligibility begins with the Committee on Special Education (CSE). 


The CSE determines, on an individual basis, whether the student will participate in: 


» the State’s general assessment with or without accommodations, 
=» the State’s alternate assessment with or without accommodations, or 


*" —acombination of the State’s general assessment for some content areas and the State’s 
alternate assessment for other content areas. 
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The CSE ensures that decisions regarding participation in the State Testing Program are not 
based on: 


"category of disability, 
» language differences, 
= excessive or extended absences, or 


=» cultural or environmental factors. 


The CSE also ensures that each student has a personalized system of communication that addresses 
his or her needs regarding disability, culture, and native language so that the student can demonstrate 
his or her present level of performance. Tests and other assessment procedures are conducted 
according to the requirements of section 200.4(b)(6) of the Regulations of the Commissioner of 
Education and section 300.320(a)(6) of the Code of Federal Regulations. 

Only students with severe cognitive disabilities are eligible for the NYSAA. The CSE determines 
whether a student with a severe cognitive disability is eligible to take the NYSAA, based on the 
following criteria: 

« the student has a severe cognitive disability and significant deficits in 
communication/language and significant deficits in adaptive behavior; and 


» the student requires a highly specialized educational program that facilitates the 
acquisition, application, and transfer of skills across natural environments (home, school, 
community, and/or workplace); and 


» the student requires educational support systems, such as assistive technology, 
personal care services, health/medical services, or behavioral intervention. 
While the State Testing Program provides full access to all students, 1% of students with severe 
cognitive disabilities in Grades 3 through 8 and in high school are alternately assessed and are counted 
as proficient for purposes of accountability. 
In accordance with 34 CFR 200.13 Adequate Yearly Progress in General, there is a 1% cap on 
the number of proficient and advanced scores on the alternate assessment that may be included in 


Adequate Yearly Progress (AYP) calculations at both the State and district levels. 


3.2 SUMMARY OF PARTICIPATION RATES 


Tables 3-1 through 3-4 show a summary of participation in the 2013-14 NYSAA by 
demographic category for each content area. 
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Table 3-1. 2013-14 NYSAA: Summary of Participation—English Language Arts 


Number Percent 


memogr apni Group Tested _ Participation 


All Students 21,737 100.00 
Male 14,839 68.27 
Female 6,898 31.73 
American Indian/Alaskan Native 178 0.82 
Black 5,457 25.10 
Asian 1,140 5.24 
Hispanic 5,659 26.03 
White 9,038 41.58 
Native Hawaiian/Other Pacific Islander 73 0.34 
Multi 192 0.88 


Table 3-2. 2013-14 NYSAA: Summary of Participation—Mathematics 


Number Percent 


Demediapii@Group Tested Participation 


All Students 21,718 100.00 
Male 14,815 68.22 
Female 6,903 31.78 
American Indian/Alaskan Native 177 0.81 
Black 5,457 25.13 
Asian 1,141 5.25 
Hispanic 5,649 26.01 
White 9,031 41.58 
Native Hawaiian/Other Pacific Islander 73 0.34 
Multi 190 0.87 


Table 3-3. 2013-14 NYSAA: Summary of Participation—Science 


Number Percent 


Pemogrephie Group Tested Participation 


All Students 9,206 100.00 
Male 6,226 67.63 
Female 2,980 32.37 
American Indian/Alaskan Native 75 0.81 
Black 2,319 25.19 
Asian 481 5.22 
Hispanic 2,355 25.58 
White 3,892 42.28 
Native Hawaiian/Other Pacific Islander 25 0.27 
Multi 59 0.64 
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Table 3-4. 2013-14 NYSAA: Summary of Participation—Social Studies 


Number Percent 


Dem OgiaphiG Croup Tested _ Participation 


All Students 3,031 100.00 
Male 1,981 65.36 
Female 1,050 34.64 
American Indian/Alaskan Native 29 0.96 
Black 731 24.12 
Asian 160 5.28 
Hispanic 696 22.96 
White 1,394 45.99 
Native Hawaiian/Other Pacific Islander 8 0.26 
Multi 13 0.43 
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CHAPTER4 TEST DEVELOPMENT 


4.1 FRAMEWORKS OF THE TESTING PROGRAM 


The New York State Common Core Learning Standards provide the framework for the New 
York State Testing Program in English Language Arts (ELA) and mathematics. The State’s core 
curriculum learning standards provide the framework for the New York State Testing Program, in 
science and social studies. Each statewide assessment program has a Test Blueprint that outlines the 
priorities to be assessed based on the grade-level learning standards. The redesign was carried out in 
response to changes that the New York State Department of Education made to the general education 
assessments to assess the Common Core Learning Standards in ELA and mathematics. The general 
education assessment Blueprints were used as the basis for the development of the alternate 
assessment Test Blueprints, which in turn drove the alternate assessment content. There is one 
alternate assessment Blueprint for each of the four content areas assessed (See Appendix A). 

In May of 2012, the New York State Education Department (the Department) assembled 
teacher committees to review the test blueprints for ELA and mathematics. The group’s goal was to 
develop Essences and Extensions for each standard. Groups focused on designing Extensions that 
aligned to general education grade-level content and, most importantly, applicability for students with 
severe cognitive disabilities. The draft Essences were reviewed by the Department and Measured 
Progress, and then posted for public comment September 10, 2012 through October 5, 2012. A total of 
852 respondents began the survey and 66.7% completed the survey. To the greatest extent possible, 
feedback collected from the survey was incorporated in to the Essence and Extension documents. 

In October 2012, the groups were reassembled with the purpose of developing Assessment 
Tasks for the approved Extensions. The draft Assessment Task documents were posted for public 
comment December 7, 2012 through January 4, 2013. A total of 1,026 respondents began the survey 
and 60.3% completed the survey. To the greatest extent possible, feedback collected from the survey 
was incorporated into the Assessment Task documents. 

The Department followed a similar process in fall 2006, when they assembled special education 
and general education teacher committees to review the core curricula and general education 
assessment Blueprints for science and social studies. This group’s goal was to determine academic 
content priorities for the NYSAA, based on the core curricula, general education assessment 
Blueprints, and, most importantly, applicability for students with severe cognitive disabilities. The 
process was designed to ensure alignment with general education grade-level content and to promote 
higher expectations for students taking the NYSAA. 
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The special education and general education teacher committees’ discussions focused on the 
actual depth and breadth of the alternate assessment requirements. Throughout the review, 
psychometricians from the Department and Measured Progress provided direction for maintaining a 
valid and reliable assessment. The resulting work by the special education and general education 
teacher committees expanded the standards for students with severe cognitive disabilities to 
Extensions for ELA and mathematics, and Alternate Grade-Level Indicators (AGLIs) for science and 
social studies. The Extensions and AGLIs provide an entry point to the grade-level content of the 
standards so that a student’s level can be gauged in terms of the standards established for all students 
by the New York State Board of Regents. 

The Test Blueprints, CCLS and core curriculum standards, Essences, AGLIs, and Assessment 
Tasks (ATs) for each grade can be found in the 2013-14 NYSAA Administration Manual: Appendix F— 


NYSAA Frameworks(www.p12.nysed.gov/assessment/nysaa/archive-12.html). 


4.2 AGLIs MAPPED TO NYS LEARNING STANDARDS AND CORE CURRICULUM BY 
GRADE 


The Extensions are aligned to the State’s Common Core Learning Standards and AGLIs are 
aligned to the New York State learning standards. Both the Extensions and AGLIs reflect high 
expectations for students with severe cognitive disabilities. This alignment is graphically illustrated in 
Figure 4-1. 

For the Extensions, a teacher committee meeting was held in October 2012 in order to review 
the new Test Blueprints for ELA and mathematics, and develop Essences and Extensions. A second 
meeting was held in May 2013 for the purpose of expanding the Extensions into Assessment Tasks. 

For the AGLIs, teacher committee meetings were held during the summer and early fall of 2006 
in order to gather input on aligning the NYSAA requirements with Grade-Level Expectations and on 
developing AGLIs. Additionally, teacher committee meetings were held in spring 2007 and 2008 to 
further refine the AGLIs and to develop additional Assessment Tasks for teachers to use in the 
alternate assessment. As part of the overall redesign implemented in 2013-14, the science and social 
studies AGLIs were narrowed and the Assessment Tasks were updated to follow the same format and 
philosophical approaches as the Extensions and Assessment Tasks in ELA and mathematics. 

The Board of Regents approved a set of learning standards to guide instruction and 
assessment. The learning standards serve as the basis of the core curricula in science and social 


studies. The curriculum of each content area is divided into the following components: 


=" science: standards and key ideas 


=" social studies: standards and units 
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Each component in a content area lists Grade-Level Expectations for student performance. These 


expectations are called grade-level performance indicators or content understandings. 
Grade-Level Expectations are further distilled into essences. Essences are the “big ideas” of the 


Grade-Level Expectations for a grade. Assessment is based on the essences for each component of 
each content area. AGLIs are aligned to the Essences in terms of three different levels of complexity 
The Board of Regents approved the New York State P-12 Common Core Leaning Standards 


(CCLS) for ELA and mathematics. The learning standards serve as the basis of the core curricula in 
ELA and mathematics. The curriculum of each content area is divided into the following components 


» ELA: strand and sub-strands 
= mathematics: domain 
Each component in a content area lists Grade-Specific Standards which are further distilled into 
Essences. Assessment is based on the essences for each component of each content area. 


Extensions are aligned to the Essences in terms of three different Levels of Complexity. 
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Figure 4-1. 2013-14 NYSAA: Mapping of Assessment Tasks to the Learning Standards 
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4.3 AGLI SELECTION CRITERIA AND PROCESS 


The New York State Board of Regents committed to the Common Core State Standards 
(CCSS) in January 2010 and formally adopted the CCSS for ELA and mathematics in July 2010. The 
New York State P-12 Common Core Learning Standards (CCLS) incorporated State-specific additions 
in January 2011. The Board of Regents announced that for students with severe cognitive disabilities, 
student progress on the CCLS would be measured, beginning with the 2013-14 administration of the 
NYSAA in ELA and mathematics. Beginning in November 2011, the Department and Measured 
Progress began developing a new test design aligned to the CCLS. In this design, new Extensions 
would be developed for ELA and mathematics, and the existing AGLIs would be refined in science and 


social studies. The process for developing the Extensions and previously the AGLIs is outlined below. 


Extensions: 

The Department coordinated the recruitment of teacher committees which met in May and 
October 2012. Participants were chosen by the Department with the intent that the participants would 
remain consistent across both meetings, which ensured consistency in the overall process and content 
interpretation. 

Participants were assigned to grade- content work groups. Each group reviewed the CCLS and 
the new Test Blueprints with the purpose of developing Essence statement(s) for each standard being 
assessed. Once the Essences were developed, the panels worked to create three Extensions for each 
standard, one for each Level of Complexity (low, middle, and most complex). The following expected 
outcomes were provided to the work groups: 

1. Each grade-content work group will produce a final draft version of an Essence statement 

that addresses the emphasis of the standard (ELA) or cluster (mathematics) and an 
Extension(s) at three complexity levels (high, medium, low) following the draft NYSAA Test 


Blueprint. The Essence and Extensions will be determined by considering: 


e Curricular congruence and alignment; 

e Developmental applicability for students with significant cognitive disabilities and 
datafolio product alignment and feasibility; 

e Applicability to transitional and career readiness skills for students with significant 
cognitive disabilities; and 

e Parental and special populations experiences ensuring consideration of all variations 


of abilities of students with significant cognitive disabilities. 
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2. Groups will have in-depth discussions between special education and general education 
teacher committees on the standards (ELA) or clusters (mathematics), Essence statements, 
and Extension statements in order to develop the final draft version of the Essences and 


Extensions. 


During an opening session facilitated by the Department and Measured Progress, participants 
were welcomed and introduced. An overview of the process and the format of the materials were 
presented. Following the opening session, participants moved into their assigned grade- content work 
groups. Using a standardized template, each group was asked to follow the same basic steps for their 


work. 


Step 1: Introductions and Material Review. 


The participants in each grade-content work group introduced themselves and indicated which 
region they were representing. A room facilitator reviewed the expectations for their work and identified 
a note taker to record key points of their discussions and decisions. Participants were asked to 
familiarize themselves with the layout of the CCLS documents and the NYSAA Alignment to CCLS 


template. 


Step 2: Develop Essence statement(s). 

Using the Alignment template as a guide, each group considered the standard being assessed 
and developed one or more Essence statements. These statements narrowed the depth and breadth of 
the content, which enables students with significant cognitive disabilities to access the content and 


demonstrate their knowledge, skills, and understanding. 


Step 3: Develop Extensions aligned to the Essence statement and transition skills. 


Using the Essence statement(s), groups developed three Extensions, which represented 
increasing complexity and cognitive demand. In addition, participants were asked to consider the 


Career Development and Occupational Studies standards and identify links to the CCLS. 


Step 4: Review the group work. 


Within each content area, work groups shared the Essences and Extensions they drafted in 
order to collaborate on the progression of knowledge, skills, and understanding. 

Following the work group meetings, an extensive review of the draft documents was conducted 
by content experts from the Department and Measured Progress. During the summer of 2012, the draft 
documents were posted on the Department’s Web site for public comment. Based on the public 
comment, additional work was done on the Essences and Extensions before they were presented to 


the work groups again in October 2012. 
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AGLIs: 

The Stakeholder groups who met in 2006, 2007, and 2008 were named the NYSAA Revision 
Workgroup (NRWG). The participants who were chosen for the initial group remained throughout all of 
the NRWG meetings, which ensured consistency in the overall process and content interpretation. 

As part of the implementation of the new test design, the Test Blueprints for science and social 
studies were revised to narrow the content assessed. In addition, minor editorial revisions were made 
to the AGLIs and Assessment Tasks. However, as was the case with the 2012-13 version of the 
NYSAA Frameworks, the intent of the AGLIs was not changed in any way. 

The spring 2008 NRWG process was consistent in science and social studies. The NRWG was 
not allowed to edit or change the Test Blueprints, Grade-Level Expectations, Essences, and intent of 
the AGLIs. As outlined below, for each content area, three steps were followed by the participants, and 


the fourth step was completed afterward by the content developers. 


Step 1: Present the expected outcomes for the work group. 


The group was welcomed and thanked for participating in the revision of the NYSAA 
Frameworks. The participants introduced themselves and indicated where they were from and in which 
content area they were participating. The presentation then consisted of directing the groups through 
the materials they would be working with and explaining the specific tasks for the grade-content work 
groups, as well as other logistical information. The group was given time for questions and then 
released into their grade-content work groups, where they remained for the rest of the day and the 


following day. 


Step 2: Review the Frameworks and other materials. 


In order to complete the tasks required in the time allotted, each content area facilitator divided 
participants into groups by grade level and distributed the materials for review. The groups were divided 


as indicated in Table 4-1. 


Table 4-1. 2013-14 NYSAA: NRWG Participant Groups from 2008 


Content Area Group Grades 
1 4 
Science 2 8 
3 High School 
; 1 5 
Social Studies 5 8, High Schoo! 
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Step 3: Complete the work process. 


In all the content area groups, the participants reviewed and edited existing Sample 
Assessment Tasks (SATs) and then worked to add new SATs. The process for adding new SATs was 
as follows: The groups first focused on AGLIs that did not have an SAT. Then they developed 
additional SATs for AGLIs that already had at least one SAT. Throughout the editing and developing of 
SATs, each group worked to ensure alignment to the AGLIs. During the editing process, the groups 
also identified words that they felt should be added to the glossary for each content area. The work 
tasks within each content area focused on each of the identified outcomes for the revision of the 
NYSAA Frameworks. 


Step 4: Review the group work as a further check on core curriculum alignment. 


Each facilitator gathered each group’s work and reviewed all edits and suggestions, as another 
check on content alignment. The edited NYSAA Frameworks then went to the Department for an 
additional content-alignment check and for finalization of each content area for the 2008-09 
administration of the NYSAA. 


4.4 ASSESSMENT TASK DEVELOPMENT 


In October 2012 the Essence/Extension work groups were reassembled with the purpose of 
developing Assessment Tasks aligned to the Extensions. Their process was similar to the steps 
followed during the May 2012 meeting. Participants were not allowed to edit or revise the Essences or 
Extensions. Using an updated Alignment template, the groups began with the first standard in their 
assigned grade-content and developed at least one Assessment Task aligned to each Extension. For 
both the Extensions and AGLIs, an Assessment Task describes an observable student action related to 
the specific knowledge, skills, and understanding aligned to the AGLI and, in turn, to the core 
curriculum. 

The 2008 NRWG developed, edited, and refined the original Assessment Tasks aligned to the 
AGLIs. Regional Lead Trainers (RLTs), who were part of the NRWG, provided input on SATs aligned to 
the AGLIs. Teachers had the opportunity to submit assessment tasks for possible inclusion in the 
NYSAA Frameworks through the annual online teacher survey. Information collected during the 2011— 
12 administration and scoring periods also influenced edits to the SATs. Edited SATs were reviewed 
and approved by the Department for the 2012-13 NYSAA Frameworks. See the following section for 
more information on task development and refer to the NYSAA Administration Manual for information 


provided to teachers regarding Assessment Task requirements. 
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CHAPTERS TEST CONTENT 


The New York State Alternate Assessment (NYSAA) is intended to provide students with severe 
cognitive disabilities the opportunity to participate in a statewide assessment that is both meaningful 
and academically challenging. Given the wide diversity of this student population, great emphasis is 
placed on ensuring that Grade-Level Expectations within the Common Core Learning Standards 
(CCLS) for English Language Arts (ELA) and mathematics and the New York State learning standards 
for science and social studies are accessible to all students. The assessment design allows students to 
demonstrate their knowledge, skills, and understanding of the CCLS through Extensions for ELA and 
mathematics and the New York State learning standards through the Alternate Grade-Level Indicators 
(AGLIs) for science and social studies. The Extensions and AGLIs are organized into three Levels of 
Complexity in order to provide an appropriate entry point for students into the standards and maintain 
the connection with the academic focus of the alternate assessment. Student performance data—Level 
of Accuracy—is collected by the teacher for each Extension and AGLI that the student is assessed 


against. 


5.1 ALTERNATE PERFORMANCE LEVEL DESCRIPTORS (APLDs) 


The Alternate Performance Level Descriptors (APLDs), developed for the standard setting that 
took place in June 2014, were used for the 2013-14 administration and reporting. Standard setting was 
conducted in June 2014 to establish cut scores for each alternate performance level in ELA and 
mathematics, Grades 3 through 8 and high school; in science, Grades 4, 8, and high school; and in 
social studies, for high school. 

The APLDs provided panelists with an idea of the knowledge, skills, and understanding related 
to the CCLS for ELA and mathematics and the core curriculum for science and social studies that a 
student at each of the four performance levels might demonstrate. A final activity during standard 
setting was for each group to provide suggestions for edits to the APLDs. The New York State 
Education Department (the Department) used the input to refine the APLDs for reporting. The APLDs 
are included in the NYSAA reports for districts, schools, parents/guardians, and educators to better 


explain each performance level. 


5.2 ACCESS TO THE GENERAL CURRICULUM 


The CCLS for ELA and mathematics contain grade-level content for pre-kindergarten through 


high school. Additionally, the core curricula for science and social studies contain grade-level content at 
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the elementary, intermediate, and secondary levels. These core curricula are aligned with the New York 
State learning standards. 

For the 2013-14 NYSAA the Department, in cooperation with teacher committees from across 
the State, has expanded the CCLS in ELA and mathematics to Extensions for students with severe 
cognitive disabilities. Previously the Department, in cooperation with Special education and general 
education teacher committees from across the state, had expanded the core curriculum Grade-Level 
Performance Expectations in science and social studies to AGLIs for students with severe cognitive 
disabilities. Extensions and AGLIs provide an entry point to the grade-level content of the standards. 
Extensions and AGLIs measure a level of mastery of the knowledge, skills, and understanding aligned 


with the CCLS and core curricula established for all students by the New York State Board of Regents. 


5.3 TEST FORMAT 


The NYSAA is a collection of student work in the form of a datafolio. The NYSAA Test 
Blueprints outline for teachers the content to be assessed at each grade and content area combination. 
The NYSAA Test Blueprints for each content area are included in Appendix A. Each of five Content 
Standards is required to be assessed for ELA and mathematics within each grade. Each of two Content 
Standards is required to be assessed for science and social studies within each grade. ELA and 
mathematics are assessed in Grades 3 through 8 and high school. Science is assessed in Grades 4, 8, 
and high school. Social studies is assessed in high school. Extensions and AGLIs are presented in the 
NYSAA Administration Manual in a spectrum of increasing complexity: low, middle, and high. 
Extensions and AGLIs must be used as written. An Assessment Task is aligned to a specific Extension 
or AGLI and describes the student action being assessed and outlines the basic expectation of what 
will be demonstrated in the verifying evidence. Teachers must use the Assessment Task as written, but 
in most cases, there is more than one Assessment Task aligned to a specific Extension or AGLI that a 
teacher may select. Allowing teachers to select an Extension and AGLI and then choose an 
Assessment Task aligned to the Extension or AGLI allows individualization while maintaining the 
content consistency of the alternate assessment. Consistency is further ensured across grade levels 
and content areas by adherence to strict administration requirements for datafolios. 

A datafolio is the resulting body of evidence across the Content Standards of a student’s 
academic performance of selected Extensions or AGLIs, as compiled by the student’s instructional 
team and scored by qualified Scorers. For each standard in ELA and mathematics, there are three 
Extensions presented across a spectrum of complexity from least to most complex. For each standard 
in science and social studies, AGLIs are presented across a spectrum of complexity from least to most 
complex. Teachers select the Extension or AGLI most appropriate for a student and plan to conduct the 
assessment. Student performance is rated by the student's instructional team according to the 


student’s Level of Accuracy in performing each Assessment Task. Two dates of student performance 
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are documented for each standard in all content areas. Teachers first administer a baseline data point 
to collect performance data and evidence that confirm that the student has not yet mastered the 
assessed skill. Based on the outcome of the baseline data point, it may be necessary to adjust the level 
of complexity (choose another task at a higher or lower level of complexity). Following the baseline data 
point, teachers provide instruction and evaluate students to gauge student growth. Before the end of 
the administration period, a final administration is collected and documented in the datafolio. In general, 
the Department recommends at least 15 school days between the baseline and final data points. To 
verify the baseline and final data points documentation, each datafolio must include verifying evidence 
that demonstrates the student’s performance of the task. Teachers may choose to submit the following: 
student work products, Data Collection Sheets, photographs, and/or digital video or audio recordings 
for the baseline administration performance and the final administration performance. Teachers 
complete the required forms and submit all documentation and evidence in a binder or fastened folder 
for regional scoring. Detailed information about the content of and procedures for developing the 


datafolio is presented in the NYSAA Administration Manual. 


5.4 ASSESSMENT DIMENSIONS 


NYSAA datafolios are scored using two dimensions: 


= Connection to Grade-Level Content 
The Connection to Grade-Level Content dimension is met when: 
o the Assessment Task is clearly aligned with the Extension or AGLI; and 


othe verifying evidence that is submitted is aligned with the Assessment Task. 
Both of the connections must be clearly evident for the Standard to be scored. 


= Performance 


o Level of Accuracy is calculated as a percentage (0%—100%). 


Level of Complexity is part of the NYSAA test design and, in addition to Level of Accuracy, factors into 


a student’s overall performance level. 
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CHAPTER6 ALIGNMENT 


6.1 PROMOTING ALIGNMENT THROUGH ACHIEVEMENT LEVEL DESCRIPTORS 


The Alternate Performance Level Descriptors (APLDs) for the New York State Alternate 
Assessment (NYSAA) are uniquely defined for each grade and content area with unifying adverbs. The 
APLDs provide a structure for understanding the knowledge, skills, and understanding that a student 
may have demonstrated in the NYSAA datafolio at a performance level. The APLDs are meant to be a 
guide or a framework to give a picture of student performance. Due to the varying abilities of students 
with severe cognitive disabilities, the APLDs were developed to be a flexible definition of student 
performance on the NYSAA. The student performance documentation that is recorded and evidenced 
within the datafolio is a more prescribed and quantified system of documentation. 

The APLDs development occurred in 2014 as part of the NYSAA redesign. The APLDs for each 
grade and content area provided panelists participating in standard setting with the official description 
of the knowledge, skills, and understanding that students are expected to display for each performance 
level. The APLDs were developed by using the old APLDs and the general education Performance 
Level Descriptors for Grades 3 through 8 in English Language Arts (ELA) and mathematics. The initial 
language was developed by the Regional Lead Trainers and was then refined by Measured Progress. 
The APLDs were reviewed, edited, and approved by the Department. 

The standard-setting panelists were able to come to a consensus with a generalized 
understanding of the terms described above due to their extensive knowledge of the NYSAA student 
population combined with understandings of the New York State Common Core Learning Standards 
(CCLS) for ELA and mathematics and the New York State core curricula for science and social studies. 
The APLDs provide information related to specific content assessed within a grade and content area 
and how that content skill may be performed by a student through his or her accuracy level. Many 
students who take the NYSAA have splinter skills, require various supports in order to perform, and can 
vary from day to day in their demonstration of the knowledge, skills, and understanding that they are 
working on within the datafolio. As such, the terms used within the APLDs provide some parameters 
and flexibility to allow for a basic picture of student performance without being specifically quantified. A 
set quantification would not allow for a representative understanding of a student with severe cognitive 
disabilities who took the NYSAA. 

Following standard setting in June 2014, the APLDs were revised by Measured Progress, based 
on input from panelists and the final recommended cut scores that were determined by panelists. The 


APLDs were then finalized and approved by the Department. 
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CHAPTER 7 ADMINISTRATION AND TRAINING 


New York State utilizes a train-the-trainer model to provide training related to the New York 
State Alternate Assessment (NYSAA). Each Board of Cooperative Educational Services (BOCES) and 
Big Five City School District designates at least one person as an Alternate Assessment Training 
Network Specialist (AATN Specialist) and at least one person as a Score Site Coordinator (SSC). (The 
Big Five City School Districts are Buffalo, New York City, Rochester, Syracuse, and Yonkers.) AATN 
Specialists and SSCs participate in the regional Administration Training conducted in September 
facilitated by the Department and Measured Progress. The AATN Specialist is responsible for 
conducting the NYSAA Administration Training with teachers. SSCs are responsible for the 
coordination of the regional Scoring Institutes; therefore, they also need to have an understanding of 


the NYSAA administration guidelines. 


7.1 STEPS FOR ADMINISTRATION 


The teacher, in coordination with the instructional team, is responsible for administrating the 
NYSAA to a student. The NYSAA Administration Manual provides detailed guidelines on how to 
administer the NYSAA to a student. The NYSAA has a specific administration period during which the 
assessment can be conducted. Assessment data cannot be collected before or after the administration 
period. The administration period for 2013-14 was September 30, 2013, to February 7, 2014. The first 
step is to review the Individualized Education Program (IEP) for a student who has been designated to 
take the NYSAA and determine the grade that the student will be assessed at, using the student's date 
of birth and the NYSAA Age Range Chart. Next, the teacher determines the Extension or Alternate 
Grade-Level Indicator (AGLI) for each Content Standard that the student will be assessed against. For 
English Language Arts (ELA) and mathematics, five standards are assessed. For science and social 
studies, two standards are assessed. Then, the teacher determines an Assessment Task that will 
demonstrate the AGLI. The Assessment Task describes the student action being assessed. Once the 
AGLIs and Assessment Tasks have been determined, the teacher conducts the Assessment Task with 
the student as a baseline administration. The baseline data point confirms that the student has not yet 
mastered the skill being assessed. If the student performance is 74% or below, then the teacher can 
continue to assess that skill. If the student performance is 75% or higher, then a higher-level skill must 
be assessed. If this is the case, the teacher would need to conduct a new baseline administration. 
Following the baseline administration, there is a period of instruction and evaluation of the skill being 
assessed. Then, the teacher conducts the Assessment Task with the student as a final administration. 
The baseline data point and final data point administrations of the Extension or AGLI and Assessment 


Task are recorded and documented. Student performance includes the student’s Level of Accuracy. 
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Verifying evidence showing the student demonstrating the knowledge, skills, and understanding of the 
Extension or AGLI through the completion of the Assessment Task must be included for the baseline 
and final administration dates of student performance documented. There are four types of verifying 
evidence that can be included, each with specific guidelines on what must be included for it to be 
considered a valid piece of evidence at scoring. The four types are student work products, a sequence 
of captioned and dated photographs, digital video or audio clips, and Data Collection Sheets. Each 
datafolio is required to have at least one Collegial Review. Collegial Review requires a colleague or 
administrator who is familiar with the NYSAA, but is not the student’s teacher who prepared the 


datafolio, to review the student's datafolio contents. 


7.2 STEPS IN CONSTRUCTING THE DATAFOLIO 


The NYSAA Administration Manual provides specific information on the construction and 
organization of the datafolio. For each Extension or AGLI, there must be a Data Summary Sheet. The 
Data Summary Sheet is the summarizing information regarding the Extension or AGLI. It includes 
student demographic information, the Extension or AGLI assessed, the Assessment Task, and student 
performance data. The baseline and final administration dates of performance data include the 
percentage for the Level of Accuracy. Also documented on the Data Summary Sheet for the baseline 
and final administrations is a yes or no to indicate if a student received verbal or physical cues or 
prompts to refocus the student on the Assessment Task. In addition to the individual requirements of 
each type of verifying evidence, the verifying evidence must include three required elements—student 
name, date of student performance, and Level of Accuracy. The teacher is responsible for ensuring that 
the verifying evidence connects to the Assessment Task and that it meets the requirements outlined in 
the NYSAA Administration Manual in order to include it in the datafolio. On or before the end of the 
administration period, the teacher assembles the datafolio in a binder or fastened folder. The datafolio 
includes a NYSAA Student Page, which provides demographic information regarding the student and 
the grade assessed, supports required per the IEP, accommodations provided during testing, and the 
month a Collegial Review was conducted. Although not required, a datafolio also includes a table of 
contents, which provides information to Scorers on where information is located in the datafolio. The 
ELA assessment documents come first, followed by mathematics, then science and social studies, if 
applicable. The Extensions or AGLIs within each content area are organized by using the numbers in 
the boxes in the upper right corner of the Data Summary Sheets (Extension 1-Extension 5; AGLI 1- 
AGLI 2). 

For ELA and mathematics, the order of documents is as follows: 


Extension 1: 


e Data Summary Sheet 
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e Verifying evidence for the baseline data point 


e Verifying evidence for the final data point 
o If either piece of verifying evidence is a Data Collection Sheet, the supporting 


evidence directly follows the Data Collection Sheet. 


This order is repeated for the remaining Extensions 2, 3, 4 and 5 in ELA and mathematics. 


Figure 7-1. 2013-14 NYSAA: Datafolio Elements for ELA and Mathematics 


ELA and 
Mathematics 


(ALL Grades) 


Extension 1 DSS Extension 2 DSS Extension 3 DSS Extension 4 DSS Extension 5 DSS 
VE for Baseline VE for Baseline VE for Baseline VE for Baseline VE for Baseline 
Data Point DEleem el iaie DEleem el iaie Data Point Data Point 


VE for 
Final Data Point 


For science and/or social studies, the order of documents is as follows: 
AGLI 1: 
e Data Summary Sheet 
e Verifying evidence for the baseline data point 
e Verifying evidence for the final data point 
o If either piece of verifying evidence is a Data Collection Sheet, the supporting evidence 
directly follows the Data Collection Sheet. 
AGLI 2: 
e Data Summary Sheet 
e Verifying evidence for the baseline data point 


e Verifying evidence for the final data point 
o If either piece of verifying evidence is a Data Collection Sheet, the supporting 


evidence directly follows the Data Collection Sheet. 
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Figure 7-2. 2013-14 NYSAA: Datafolio Elements for Science and Social Studies 


Science and Social 
Studies 


(Grade specific) 


AGLI 1 AGLI 2 


Data Summary Data Summary 
Sheet Sheet 


VE for Baseline VE for Baseline 
Data Point Data Point 


VE for Final Data VE for Final Data 
Point Point 


7.3 ADMINISTRATION TRAINING AND COLLEGIAL REVIEW 


In September 2013, the Department, in collaboration with Measured Progress, trained AATN 
Specialists and SSCs from across the state on how to conduct the NYSAA Administration Training with 
teachers in their regions. The one-day trainings were conducted regionally across the state over a two- 
week period. There were three main activities conducted. First, information regarding updates to the 
NYSAA and the materials was provided. Then, the NYSAA Administration Training DVD was shown, as 
well as completion and review of the Guided Practices. Last, the participants were asked to work in 
groups to discuss strategies to help teachers with the changes and develop some tips for 
understanding the transition to the 2013-14 NYSAA. 

A total of five NYSAA Administration Trainings occurred at four geographically diverse sites: the 
Albany region, which included Long Island and the regions surrounding New York City; the Syracuse 
region; the Buffalo and Rochester region; and the New York City region, which included the non-District 
75 trainers on one day and the District 75 trainers on another day. Table 7-1 outlines the number of 
participants at each training session. 


Table 7-1. 2013-14 NYSAA: Administration Updates Training—Participant Count 


Buffalo- | New York City 


mIBanY < Syraetse Rochester Region(Two Total 


Region Region 


Region Trainings) 
NYSAA 
Administration 61 25 51 121 258 
Updates Training 


Administration Training for teachers is provided through a combination of Guided Practices and 
a training DVD. AATN Specialists are required to use all parts of the DVD and Guided Practices, as 
specified by the Department. The NYSAA Administration Training DVD is organized into segments. 
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There is an Opening segment, a Department Messages segment, a Steps for Administration segment, 
and a Recommendations and Closing segment. The opening segment provides general information 
about what is going to be covered during the training session. The Department messages segment 
provides the background for the implementation of the new NYSAA test design and alignment of ELA 
and mathematics to the CCLS. The steps for administration segment is a detailed review of each of the 
steps for administering the NYSAA. The detailed information includes things to consider in planning for 
the assessment, specifics regarding administering the assessment, and an outline of steps for 
assembling and submitting the datafolio for scoring. The information provided in this segment follows 
the organization of the NYSAA Administration Manual, and includes many visuals to assist teachers in 
understanding the NYSAA. The Recommendations and Closing segment provides best practices tips 
and strategies on how to maintain the Connection to Grade-Level Content during administration, 
information on prompts and cues, keys changes in the assessment design, and things to keep in mind 
for the 2013-14 NYSAA, as well as next steps for teachers and information regarding Collegial 
Reviews. At specific points throughout the segments, there are stop points built in, and a Guided 
Practice must be conducted at each of these points. The Guided Practices reinforce the information 
that was contained in the segment. There are a total of four Guided Practices. The first Guided Practice 
focuses on understanding how to determine the correct grade a student should be assessed at based 
on his or her date of birth, and how to use the Test Blueprints in the NYSAA Administration Manual 
Appendix F: NYSAA Frameworks (http://www.p12.nysed.gov/assessment/nysaa/nysaa-manual- 
14.html). The second Guided Practice focuses on understanding how to navigate through the NYSAA 
Frameworks in order to select the Extensions and AGLIs and to determine some verifying evidence 
options for specific Extensions and AGLIs. The third Guided Practice focuses on determining and 
documenting baseline student performance and determining if the baseline threshold is exceeded. The 
fourth Guided Practice provides teachers with a general NYSAA requirements review worksheet. 
Teachers complete all four Guided Practices, and a review of the practices is facilitated by the AATN 
Specialists. At or before the locally conducted NYSAA Administration Trainings, teachers are provided 
with the NYSAA Administration Manual, which includes the NYSAA Frameworks as Appendix F. 
Collegial Review is required for each student datafolio. Collegial Review is meant to be an 


independent review of a datafolio. The Department provides guidance that reviewers should: 


=" be familiar with the current alternate assessment; and/or 


» have attended 2013-14 Administration training in the fall of 2013. 


The Department guidelines recommend that Collegial Reviews take place during the planning 
phase, at a midpoint during administration, and prior to the end of administration. The teacher is given 
feedback about whether the appropriate connections have been made between the Extensions or the 


AGLIs and the Assessment Tasks and between the Assessment Tasks and the verifying evidence. 
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Also, Collegial Reviews help to confirm that all documents included in the datafolio at that point meet all 
procedural requirements. The Department cautions that a Collegial Review helps ensure, but does not 


guarantee, that a datafolio meets the procedural requirements in order for a student to receive a 
reportable score. 
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CHAPTER8& SCORING 


Alternate Assessment Training Network Specialists (AATN Specialists) and Score Site 
Coordinators (SSCs) participate in the regional Scoring Training conducted each year. SSCs are 
responsible for the coordination of the regional Scoring Institutes, and must pass the qualification 
samples in order to make content decisions during the Scoring Institute. The AATN Specialist acts as a 
Floor Manager at a Scoring Institute, and must also pass the qualification samples in order to make 
content decisions during the Scoring Institute. 

In February and March 2014, the Department, in collaboration with Measured Progress, trained 
AATN Specialists and SSCs from across the state on how to score New York State Alternate 
Assessment (NYSAA) datafolios and how to conduct the NYSAA Scoring Training with Scorers at the 
Scoring Institute in their region. The one-day trainings were conducted regionally across the state over 
a two-week period. There were three main activities conducted. First, information regarding updates to 
the NYSAA Scoring Procedures and Decision Rules and the materials was provided. Then, the NYSAA 
Scoring Training DVD was shown, as well as completion and review of the practice samples. Last, the 
participants were asked to complete the qualification samples. Retraining and qualification was 
provided as needed. 

A total of five NYSAA Scoring Trainings occurred at four geographically diverse sites: the 
Albany region, which includes Long Island and the regions surrounding New York City; the Syracuse 
region; the Buffalo and Rochester region; and the New York City region, which includes the non-District 
75 trainers in one training session and the District 75 trainers in another training session. Table 8-1 


outlines the number of participants at each training session. 


Table 8-1. 2013-14 NYSAA: Scoring Training—Participant Count 


Buffalo- | New York City 
Rochester Region (Two Totals 
Region Trainings) 


54 31 36 116 237 


Albany Syracuse 
Region Region 


NYSAA Scoring 
Training 


8.1 SCORING OF OPERATIONAL TESTS 


The scoring of NYSAA datafolios occurs during the spring, following the close of the 
administration period. Scoring is a decentralized process carried out at regional Scoring Institutes. The 
Department provides a scoring window within which the institutes conduct their scoring sessions. The 
purpose of the Scoring Institute is to provide a forum in which educators individually score the NYSAA 


student datafolios. Each Scoring Institute is overseen by an SSC and an AATN Specialist. These 
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individuals are thoroughly trained and participate in a qualifying process conducted by the Department 
and Measured Progress. They are each given a duplicate set of training materials that are to be used 
during turnkey training at their own Scoring Institutes. They are required to follow the model of the 
training process demonstrated by the Department and Measured Progress. 

There are a variety of processes involved in the Scoring Institute. The basic outline for the 
review of student datafolios consists of three major steps. Scorers review student datafolios; confirm 
that the Connection to Grade-Level Content, including the baseline data point performance information, 
is satisfied; and verify the percentages and ratings for Accuracy and Independence documented by the 
teacher for each Extension or Alternate Grade-Level Indicator (AGLI) assessed. Any questions that 
arise during scoring are directed to a Table Leader. Scorers use the document entitled Steps for 
Scoring 2013—14 NYSAA Datafolios as the main reference for scoring each datafolio. Table Leaders 
use the Decision Rules for Scoring 2013-14 NYSAA Datafolios as a reference document for any 
questions that are not addressed in the Steps for Scoring 2013-14 NYSAA Datafolios. Both documents 
are included in this report, as Appendices B (Scoring Procedures) and C (Scoring Decision Rules). 

On a worksheet, a Scorer records the Extension or AGLI code, Connection to Grade-Level 
Content questions, percentages for the Level of Accuracy for the baseline administration and final 
administration, a question about whether the student was prompted, and Scorer comments. Part of this 
worksheet is returned to the school district along with the datafolio for review by the instructional team 
and administrators. 

Once a datafolio has been reviewed completely, the last step is for the Scorer to transcribe the 
Extension or AGLI codes, Connection to Grade-Level Content questions, percentages, and other 
information onto a Scannable Score Document (Scannable). The score document is scanned by the 
Regional Information Center (RIC) or the Big Five City Scan Center. (The Big Five City School Districts 


are Buffalo, New York City, Rochester, Syracuse, and Yonkers, each having its own City Scan Center.) 


8.2 SCORING RUBRIC 


The Scoring Rubric is the initial guide that drives the model used to score NYSAA datafolios. 
The Scoring Rubric is provided in the 2013-14 NYSAA Administration Manual, along with guidance on 
the process that teachers must follow in order to meet the scoring requirements. The rubric is broken 
into two parts. The first part outlines the grades and content requirements, and provides some brief 
assessment requirements information. The second part provides information about the factors for a 
performance level. The factors included are the Connection to Grade-Level Content, student 
performance, and Level of Complexity. The Connection to Grade-Level Content is explained on the 
Scoring Rubric as follows: “Extensions/AGLIs are assessed based on the appropriate grade level 
academic content for students with severe cognitive disabilities. The Assessment Task must align to 


the Extension/AGLI chosen AND the verifying evidence must be aligned to the task. If these 
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connections are not clear, the Extension/AGLI will not be scored.” The final administration Level of 
Accuracy provides the percentage for the performance dimension. For each Assessment Task 
documented, the percentage for Level of Accuracy (relative to the student’s demonstration of skills, in 
relation to the Extension or AGLI) and the Level of Complexity that the Extension or AGLI came from 


combine to give the overall performance level. The Scoring Rubric is presented in Table 8-2. 
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Table 8-2. 2013-14 NYSAA: Scoring Rubric 


Students with disabilities participating in the NYSAA are assessed according to chronological ages aligned to grade levels. Refer to the Age Range Chart 
for current date of birth ranges. Students should be tested only once at each grade and in all the content areas indicated for each grade. For all content 
areas, Student performance data is collected on at least two dates within the administration period. Baseline data must be collected to confirm the student 
has not yet mastered the selected Extension or AGLI. 

Grade ELA Mathematics Science Social Studies 


3 5 Standards 5 Standards 
4 5 Standards 5 Standards 2 Standards 
5 5 Standards 5 Standards 
6 5 Standards 5 Standards 
7 5 Standards 5 Standards 
8 5 Standards 5 Standards 2 Standards 
High School 5 Standards 5 Standards 2 Standards 2 Standards 
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Table 8-3. 2013-14 NYSAA: Factors for a Performance Level: Connections to Grade-Level Content, Performance, Level of Complexity 


Connection to Grade-Level Content = Extensions/AGLIs are assessed based on the appropriate grade level 
academic content for students with severe cognitive disabilities. The Assessment Task must align to the 
Extension/AGLI chosen AND the verifying evidence must be aligned to the task. If these connections are not clear, 
the Extension/AGLI will not be scored. 


Connection to Grade-Level Content Progression: 


Extension/AGLI from 


feye-\e (= 


Assessment Task 
EVI Colal-xe mm ve) 


Extension/AGLI 


Verifying Evidence 


aligned to Assessment 
Task 


Performance = Level of Accuracy (%) 


Level of Accuracy 


Independence 


The student demonstrates skills based on the Extensions or AGLIs resulting ina 
percentage for Level of Accuracy. 

Was the student prompted in any way during the administration of the Assessment 
Task? Yes or No. 


*NEW* 
Level of Complexity 


Less Complex 


Middle 


Most Complex 


Required Standard not 
assessed 

Extension or AGLI assessed 
from incorrect grade 
Incorrect Assessment Task 
assessed 

Verifying evidence does not 
demonstrate task 


Required data points and/or 
evidence not submitted 
Required elements not 
documented on evidence 
Verifying evidence not valid 


No or No Score (NS) results when one or more of these issues are identified during scoring (including but 
not limited to) 


Connection to Grade-Level Content Performance Level of Complexity 


Score for baseline 
administration over 
threshold (Level of 
Accuracy is 75% or higher) 
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8.3 SCORING PROCESS AND RELIABILITY MONITORING REVIEW 


8.3.1 Scoring Process 


Scorers, who are all New York State teachers or other licensed and/or certified professionals, 
are directed to objectively review and document the ratings for student performance data contained in 
the datafolio. During the Scoring Training, it is explained that the data provide an opportunity for 
students to demonstrate their knowledge, skills, and understanding of the grade-level content. Scoring 
processes are consistent from one grade level to the next. The same procedures and rules apply to all 
grade levels and content areas, which is critical to the procedural validity of the assessment. 

Scoring Training includes a DVD presentation, a series of practice samples, and Scorer 
qualification. (These are described in further detail in the next section.) 

The actual scoring process involves reviewing the datafolio compiled by the teacher. The review 
is meant to ensure that all requirements are met. The Scorer records the rubric rating for each AGLI 
assessed. If the Connection to Grade-Level Content and the baseline administration performance is 
satisfied, the final performance percentages can be confirmed, and each performance percentage for 
baseline and final administrations can be recorded by the Scorer. If the Connection to Grade-Level 
Content is not met or the baseline administration performance is above the percentage threshold, a 
rating of No Score (NS) is recorded. After the Scoring Institute, the Scorer ratings are converted to the 
alternate assessment performance levels, which appear on the NYSAA reports. 

In order for Scorers to complete their review of the datafolios, a set of standardized tools is 
provided to each Scoring Institute. These tools include the NYSAA Administration Manual and 
Frameworks, Scoring Procedures, Scoring Decision Rules, Guided Practices, and qualifier sets. 
Student performance ratings are documented on a Scorer Worksheet with a Menu of Comments and a 
Scannable. The Menu of Comments, located on the back of each page of the Scorer Worksheet, 
includes information that a Scorer records when an Extension or AGLI has a No Score rating. It also 
allows the Scorer to provide additional constructive feedback to a teacher about the datafolio. 

There are 14 steps involved in the scoring process. The step-by-step procedures outlined in the 
Steps for Scoring 2013-14 NYSAA Datafolios are implemented statewide, and ensure scoring reliability 


across all Scoring Institutes. Table 8-4 presents a quick review of the steps. 
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Table 8-4. 2013-14 NYSAA: Scoring Steps Quick Reference 


Step/(s) 
1 Student demographics, Scorer ID, Scoring Institute code, Confirm student’s date of birth and grade 
level assessed, Testing Accommodations, and Collegial Review 
2aandb Review sequence of documentation for content area 
3 Demographic information on DSS complete and accurate when compared to the Student Page 
4 Extension or AGLI from grade level (Connection to Grade-Level Content) 
5 Task connects to Extension or AGLI (Connection to Grade-Level Content) 
6a, b, Verifying evidence connects to task (Connection to Grade-Level Content) and Level of Accuracy for 
and c the baseline data point is 74% or below 
7 Dates on DSS within the administration period 
8a-f Valid verifying evidence and supporting evidence 
a_ Valid verifying evidence and supporting evidence: Required elements clearly documented (3) 
b Valid verifying evidence: Student Work Product: Original 
c_ Valid verifying evidence: Data Collection Sheet (DCS): Minimum of three dates, includes supporting 
evidence and staff initials 
d_ If verifying evidence is DCS, supporting evidence is present and valid 
e Valid verifying evidence: Photographs: Minimum of three sequential, captioned, and dated 
photographs 
f Valid verifying evidence: digital video or audio clip: clip is brief and has recorded markers 
9 Supports provided that guided the student to the correct answer 
10 Confirm final administration percentage for Level of Accuracy, record percentages for Level of 
Accuracy for final and baseline administrations, record if student was prompted for final and baseline 
administrations 
A Score the second Extension or AGLI (Steps 3-10) 
12 Score mathematics, science, and social studies (Steps 2-11) 
13 Confirm Scorer Worksheet is complete, including Procedural Error Comments and additional Scorer 
Comments 
14 Complete the Scannable Score Document 


The Scoring Procedures are separated into two major sections: preparing to score and 


reviewing and scoring a datafolio. Each step asks the Scorer a question or directs the Scorer to confirm 


a certain requirement. The steps are presented in a yes/no format to assist the Scorer in moving from 


one step to another. If a Scorer encounters a “no” or an issue outside the directions provided in the 


Scoring Procedures, he or she must consult with the Table Leader. The Table Leader refers to the 


Decision Rules for Scoring 2013-14 NYSAA Datafolios, if the information on how to proceed in scoring 


the datafolio is not already provided in the Scoring Procedures. 


The Scoring Decision Rules have their own segment in the training DVD. There is also a brief 


overview of the Decision Rules within the Scoring Procedures segment of the training DVD. The 


Decision Rules serve as guidance for Table Leaders when a Scorer encounters an issue that is outside 


the direction provided in the Scoring Procedures document. The rules are organized by topic, beginning 


with rules that apply to the datafolio as a whole (e.g., incorrect forms, missing Student Page, evidence 


of photocopies, correction fluid/tape or black out). The other topic headings are “Assessment Task,” 


“Verifying Evidence,” and “Dates.” Fifteen Decision Rules were developed that are based on actual 


datafolio issues found during a Benchmarking review of datafolios in progress. In the training DVD, 
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Scoring Decision Rules are presented by number as found in the Decision Rules chart. If possible, an 
example is provided, that highlights the point of the Decision Rule, and a description is provided 


regarding how the rules are to be consistently applied statewide at each Scoring Institute. 


8.3.2 Reliability Monitoring Review 


The purpose of the Reliability Monitoring Review (RMR) is to ensure scoring consistency and 
reliability across Scoring Institutes. 

At the end of the Scoring Institute, 20% to 25% of the scored datafolios from each scoring site 
are randomly collected by the SSC for the RMR. The range of 20—25% applies only to the 2013-14 
NYSAA administration, and was done as a means to collect enough datafolios that could be used for 
the standard setting that took place in June 2014. Measured Progress conducts a Scoring Institute in 
which the random datafolios are scored by highly experienced and qualified Scorers. RMR Scorers 
complete the same NYSAA training and qualification process that is used statewide. 

RMR scores are compared with the original scores from the regional Scoring Institutes. The 
original score remains the score of record; the RMR score does not change or affect the original score 


in any way. The 2013-14 RMR results are presented in Chapter 10. 


8.4 SCORER QUALIFICATION AND TRAINING 


A standardized statewide process for Scorer Training and qualification is observed. Each Board 
of Cooperative Educational Services (BOCES) and Big Five City School District conducts at least one 
two-day Scoring Institute during the scoring period. For 2013-14, the scoring period was March 10— 
May 1, 2014. The same training and scoring process, Scoring Procedures, and Decision Rules were 
applied and implemented statewide. 

The DVD presentation portion of the training includes a welcome and introduction, which briefly 
outlines the DVD segments and documents used during training. The DVD then outlines the scoring 
tools, the step-by-step process for reviewing the datafolios and documenting student scores, and the 
practice scoring that is done while following along with the DVD segment. The first practice is 
completed according to directions outlined in the DVD segment. The first Extension is completed as 
part of the DVD segment. The DVD segment is then paused to complete the second Extension. The 
second Extension is completed as a group or in pairs. The DVD segment provides details about how 
the second Extension should have been scored. 

After the first two DVD segments, Scorers practice scoring two additional datafolio samples with 
two Extensions or AGLIS—first as a group or in pairs, and then individually. Each practice is reviewed 
to ensure that Scorers are following the Scoring Procedures accurately. The final DVD segment details 


other best practice information for scoring, and reinforces information about confirming connection of 
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the verifying evidence to the Assessment Task and Data Collection Sheets. It also provides details 
about the subsequent steps in Scorer Training. 

After the DVD, Scorers are given an opportunity for final questions. Training ends with Scorers 
completing three calibrated qualifiers with two Extensions or AGLIs each. The qualifiers are actual 
student datafolios in a content area. The qualifiers were identified by a group of Special education and 
general education teacher committees during a Benchmarking process. Each Scorer must earn a score 
of 80% or higher to become qualified. Scorers who do not qualify on the first qualifier set receive 
additional training and must complete an additional qualification sample. After the initial set, Scorers 
have three opportunities to receive retraining and to qualify. If a Scorer does not qualify after additional 


attempts, he or she is reassigned to another role in the Scoring Institute. 


8.5 SCORING QUALITY CONTROL 


The quality control process at each Scoring Institute is handled by the SSC, AATN Specialists, 
and Table Leaders. The SSC is responsible for planning, conducting, and coordinating NYSAA scoring 
activities for the regional Scoring Institute. Each BOCES or Big Five City School District designates at 
least one individual to assume the role of SSC. 

SSC responsibilities include: 

"ensuring that the Scoring Procedures, Decision Rules, and other scoring-related 
guidelines are implemented consistently per the Department’s prescribed model; 


* ensuring the lock-and-key security of all datafolios during storage and throughout all 
scoring sessions (datafolio security must be maintained throughout this process); 


"gathering the NYSAA student registration information from the RIC or Big Five City Scan 
Center to assist in planning the Scoring Institute; 


# planning, coordinating, and conducting the Scoring Institute for each BOCES or Big Five 
City School District; 


" being present at all times while scoring is in session; 


* coordinating the selection of sample datafolios as requested by the Department for 
RMR; 


* ensuring that scoring documentation is completed and provided to the RIC or Big Five 
City Scan Center; 


* collecting feedback regarding the Scoring Institute from AATN Specialists, Table 
Leaders, and Scorers; 


= providing feedback to the Department about the scoring process, procedures, and 
documentation; and 


* returning datafolios following scoring. 
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AATN Specialists are designated by each BOCES or Big Five City School District to conduct 
information sessions and NYSAA training and to assist with scoring. 
For NYSAA scoring, AATN Specialists: 


" assist SSCs in the planning of the Scoring Institute, as needed; 


" conduct training sessions and facilitate qualification sessions for Table Leaders and 
Scorers; 


"act as Floor Managers during the scoring process; 

» resolve Table Leader questions, using scoring guidelines and resources; 

"participate in the Read Behind Process; and 

" provide feedback to SSCs and the Department about the scoring processes, 
procedures, and documentation. 

Table Leaders are integral to making sure that the processes and procedures outlined by the 
Department in the Scoring Training are followed at each scoring station during each Scoring Institute. 
There is one Table Leader for every five Scorers. 

For NYSAA Scoring, Table Leaders must: 


= be experienced Scorers familiar with the 2013-14 NYSAA; 


"complete Scoring Training, including the qualification process, prior to the start of the 
Scoring Institute; 


"coordinate the datafolio flow at their assigned scoring stations; 

» resolve questions from Scorers, using scoring guidelines and resources; 

= review and confirm all adjustments and all No Scores documented by Scorers; 
"conduct quality control checks of scored datafolios; 

=" manage the Read Behind Process; 

"separate copies of the Scorer Worksheet as designated by the SSC; 

* return scored datafolios to the appropriate box; 


"provide feedback to the SSC and the Department about the scoring process, 
procedures, and documentation; and 


" assist as needed in evaluating and providing additional training for Scorers who do not 
qualify during the first round of qualifying. 


The Table Leaders are responsible for three main quality control checks. Their first responsibility is to 
resolve Scorer questions and to confirm NS ratings. When a Scorer questions the Connection to 
Grade-Level Content or has a question about scoring a datafolio that may result in an NS, the datafolio 
must be reviewed with the Table Leader. If the issue cannot be readily resolved by the Table Leader by 


using the Scoring Procedures and Scoring Decision Rules, it must be brought by the Table Leader to 
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the Floor Manager. If the issue cannot be readily resolved by the Floor Manager, the SSC will make the 
final decision. 

The second responsibility of a Table Leader is to complete a standardized quality control check. 
A quality control check is conducted by the Table Leader once a datafolio has been scored and 
returned by a Scorer. The Scorer Worksheet is cross-checked against the Scannable. Any corrections 
made to the ratings by the Scorer are double-checked, and comments are confirmed as being 
appropriate. A blue dot is affixed by the Table Leader to confirm that the quality control check was 
conducted. 

The third responsibility of a Table Leader is to manage the Read Behind Process. The Read 
Behind Process occurs throughout the Scoring Institute. This process ensures the integrity of scoring 
across scoring stations. Table Leaders select the first, third, and then every seventh datafolio from each 
Scorer for a read behind. The Scannable is pulled and held by the Table Leader, and a red dot is 
placed on the datafolio. This indicates that it has been selected for a read behind. The first Scorer 
scores the datafolio, completes the Scorer Worksheet, and returns the datafolio to the Table Leader. 
The Table Leader turns the Scorer Worksheet over, places it into the front pocket of the datafolio, and 
then routes the scored datafolio to be scored at a different scoring station or a read-behind table for the 
second read. The second Scorer scores the datafolio, completes a second Scorer Worksheet, and 
returns the datafolio to the Table Leader. The Table Leader (either at the first scoring station or read 
behind table) compares the two worksheets. If no discrepancy exists, the Table Leader fills in his or her 
Scorer ID# and completes the Scannable. A quality control check is completed, and a blue dot is affixed 
to the datafolio. The second Scorer Worksheet is destroyed. If a discrepancy between the scores is 
found, the Table Leader highlights the discrepant areas and forwards the datafolio to the Floor Manager 
or SSC for resolution. The Floor Manager or SSC reviews the discrepant areas, enters his or her 
Scorer ID#, and completes the Scannable. The Floor Manager returns the datafolio to the Table Leader 
for quality control. After a datafolio has been through the Read Behind Process, the Table Leader 
completes a quality control check. The Table Leader then works with the Scorer to review the 
discrepancy and provides any training or support that is needed. If the Scorer continues to have 
discrepant scores, the Table Leader is then directed to consult the Floor Manager and/or the SSC to 
discuss additional training or reassignment. 

As an additional quality control check to confirm that the Scoring Institutes are following all the 
processes and guidelines prescribed by the Department, a score site observation visit is conducted on 
a sample of Scoring Institutes. Each year, the Department designates a set of sites to be monitored 
during their Scoring Institutes. The observation visits are conducted by the Regional Lead Trainers 
(RLTs). SSCs are notified if they are selected by the Department for observation. Observers cannot 


participate or assist in any part of the Scoring Institute. They cannot interact or provide technical 
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assistance during the observation. An Observation Protocol Checklist is completed during the visit and 


submitted to the Department. 
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CHAPTER9Y CLASSICAL ITEM ANALYSIS 


As noted in Brown (1983), “A test is only as good as the items it contains.” A complete 
evaluation of a test’s quality must include an evaluation of each item. Both the Standards for 
Educational and Psychological Testing (AERA, APA, NCME, 2014) and the Code of Fair Testing 
Practices in Education (Joint Committee on Testing Practices, 2004) include standards for identifying 
quality items. While the specific statistical criteria identified in these publications were developed 
primarily for general—not alternate—assessment, the principles and some of the techniques apply 
within the alternate assessment framework, as well. 

Both qualitative and quantitative analyses were conducted to ensure that New York State 
Alternate Assessment (NYSAA) items met these standards. Qualitative analyses are described in 
earlier sections of this report; this section focuses on the quantitative evaluations. The statistical 
evaluations discussed are difficulty indices, discrimination (item-test correlations), item means, 
structural relationships (correlations between the dimensions), and bias and fairness. The item 


analyses presented here are based on the statewide administration of the 2013-14 NYSAA. 


9.1 DIFFICULTY AND DISCRIMINATION 


For the NYSAA, each student datafolio for a specified content area at a given grade level 
receives an Accuracy score on each of five standards in English Language Arts (ELA) and 
mathematics, and on each of two standards in science and social studies. For each standard, teachers 
choose a task by which to assess their students. The chosen task may be at one of three possible 
Levels of Complexity (LOC: LOC1, LOC2, or LOC3), where the higher levels indicate greater 
complexity. For a given student, the LOC at which the student is assessed may differ from one 
standard to another. Thus, for any one standard, the number of students assessed varies across the 
LOCs, and the way the student counts vary across the LOCs varies across the standards. Tables H-1 
to H-18 in Appendix H, Classical Item Analysis, include the student counts for the three LOCs for each 
assessed standard. Table 9-1 below summarizes the means and ranges of these counts. As can be 
seen in these tables, approximately 3000 students were assessed at each grade level; on average, 
about 64% of the students were assessed at LOC1, about 29% at LOC2, and about 7% at LOC3. Note, 
however, that there also existed substantial variability across standards in how students were 
distributed across the LOCs. For example, for Grade 3 ELA, for Standard 314 about 66% of the 
students were assessed at LOC2, whereas for Standard 322 about 83% were assessed at LOC1. In 
general, for every grade level and content area, LOC1 and LOC2 combined to easily have the most 
assessed students. The largest count and percentage of students assessed at LOC3 occurred for 


Standard 911 of high school mathematics, where the 638 students were only 21% of the total. 
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Table 9-1. 2013-14 NYSAA: Summary of Numbers of Students Assessed Across the Levels of Complexity 


Subject Gade LOC 1 LOC 2 LOC 3 
Mean Min Max Mean Min Max Mean Min Max 
3 1550.0 763 2172 906.8 383 1755 182.4 56 290 
4 1912.0 1302 2413 814.6 327 1473 234.6 46 497 
English 5 2066.4 1663 2539 835.8 433 1300 128.6 29 321 
Language 6 2016.2 1645 2505 1069.4 612 1549 143.8 20 393 
Arts 7 2565.6 2272 2839 466.8 212 703 213.8 55 422 
8 2439.8 1855 2651 506.4 329 1014 130.8 75 204 
High School 1708.6 1426 2209 1076.4 575 1385 171.6 87 270 
3 1492.6 964 2219 996.6 244 1623 152.8 48 269 
4 2121.8 1751 2301 536.6 240 634 323.2 31 610 
5 2301.2 1715 2609 561.8 298 1228 155.6 50 361 
Mathematics 6 2208.8 1866 2633 861.6 499 1289 152.0 50 306 
7 2003.8 1221 2692 950.8 301 1855 300.2 113 555 
8 2405.8 2043 2979 510.6 90 832 167.4 45 431 
High School 1698.6 1311 2080 853.2 627 1541 401.6 95 638 
4 2021.5 1822 2221 771.0 561 981 167.0 141 193 
Science 8 1548.0 1431 1665 1255.5 1154 1357 264.0 239 289 
High School 1408.5 1276 1541 1266.5 1040 1493 296.5 193 400 
Social Studies High School 1121.0 1106 1136 1748.0 1735 1761 112.5 83 142 


For each task, the teacher assessed each student a certain number of times (determined by the 
teacher), and the teacher recorded the number of times the student was successful. Because the 
number of times a student was assessed for each task was something teachers could decide for 
themselves, this number varied across students. Therefore, a percent-correct score was recorded 
rather than number correct score. Hence, for a standard assessed at a particular LOC, the observed 
student scores ranged from 0 to 100. 

To develop a single scale for comparing all students for a specific standard, percent-correct task 
scores associated with higher LOCs deserve more credit than the same scores at lower LOCs. Based 
on a scientific study jointly carried out by Measured Progress and the Department, a single scale was 
developed that allows scores at different LOCs for a given standard to be combined into a single scaled 
score for that standard. Specifically, the scaled score for a given standard is calculated by taking the 
observed percent-correct score, adding a credit when the LOC is a 2 or 3, and then dividing the result 
by 10. The complexity credit was 75 for an LOC of 2, and 150 for an LOC of 3. The division by 10 was 
needed to eliminate unwanted gaps in the scale, resulting from some combinations of scores being 
much less likely than others. Thus, the scaled scores for a given standard could range from 0 to 25. 
Thus, the formulas for the individual standard scaled scores by complexity are as follows: 

SSioc1 = RS,0c1/10 
SSioca = (RSzocz + 75)/10 
SSioc3 = (RSz0c3 + 150)/10 
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To compare students at the level of total test score, a scaled score that is simply the sum of the 
scaled scores on the standards plus an additive constant is produced. Specifically, for ELA and 
mathematics, total score is the sum of the scaled scores on the five standards plus 400, resulting in 
total scores that can range from 400 to 525. And for science and social studies, the total score is the 
sum of the scaled scores on the two standards plus 550, resulting in total scores that can range from 
550 to 600. Thus, the formulas for the total subject standard scaled scores are as follows: 

SStotal ELA/Math = SSstai + SSstaz + SSsta3 + SSstaa + SSstas + 400 
SStotat sci/ss = SSsta1 + SSstaz + 550 

From the above, it is clear that there are two types of scores on this test that could be treated as 
traditional “item” scores for the purposes of psychometric evaluation. The standards are one 
reasonable choice to represent the traditional items on the test because each student is assessed on 
the same number of standards, and the sum of the scaled scores they receive on the standards 
provides the basis for the total scaled score for a student. Alternatively, the application of a standard at 
a given LOC (S/LOC) could also be used to represent a traditional item because the raw percent- 
correct scores are directly comparable across students who were assessed on the same standard at 
the same LOC. 

Using both of these item representations (“Standard” and “S/LOC”), all items were evaluated in 
terms of item difficulty according to standard classical test theory practices. “Difficulty” was defined as 
the average proportion of points achieved on an item and was measured by obtaining the average 
score on an item and dividing by the maximum score for the item. By computing the difficulty index as 
the average proportion of points achieved, the items are placed on a scale that ranges from 0.0 to 1.0. 
Although the p-value is traditionally described as a measure of difficulty (as it is described here), it is 
properly interpreted as an easiness index, because larger values indicate easier items. 

An index of 0.0 indicates that all students received no credit for the item, and an index of 1.0 
indicates that all students received full credit for the item. Items that have either a very high or very low 
difficulty index are considered to be potentially problematic because they are either so difficult that few 
students get full credit or so easy that nearly all students get full credit. In either case, such items 
should be reviewed for appropriateness for inclusion on the assessment. 

It is worth mentioning that using a norm-referenced criterion such as p-values to evaluate test 
items is somewhat contradictory to the purpose of a criterion-referenced assessment like the NYSAA. 
Criterion-referenced assessments are intended primarily to provide evidence of student progress 
relative to a standard rather than to differentiate between students. Thus, the generally accepted 
criteria regarding classical item statistics are only cautiously applicable to the NYSAA. 

A desirable feature of an item is that higher-ability students perform better on the item than do 


lower-ability students. The correlation between student performance on a single item and total test 
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score is a commonly used measure of this characteristic of an item. Within classical test theory, this 
item-test correlation is referred to as the item’s “discrimination” because it indicates the extent to which 
successful performance on an item discriminates between high and low scores on the test. The 
discrimination index used to evaluate NYSAA items was the Pearson product-moment correlation. The 
theoretical range of this statistic is -1.0 to 1.0. 

Discrimination indices can be thought of as measures of how closely an item assesses the 
same knowledge and skills assessed by other items contributing to the criterion total score. That is, the 
discrimination index can be thought of as a measure of construct consistency. In light of this 
interpretation, the selection of an appropriate criterion total score is crucial to the interpretation of the 
discrimination index. For the NYSAA, the total test scaled score, excluding the item being evaluated, 
was used as the criterion score. 

A summary of the item difficulty and item discrimination statistics for each grade/content area 
combination is presented in Tables 9-2 and 9-3 for the two kinds of items discussed above, “S/LOC” 
and “Standard,” respectively. As shown in Table 9-2, the mean difficulty values for S/LOC items ranged 
from 0.81 to 0.89, indicating that, overall, students performed well on the S/LOC items on the NYSAA, 
and that students, on average, were well prepared for the LOCs that were targeted by their instruction 
for each standard. On the other hand, as shown in Table 9-3, the mean difficulty values for the 
Standards-based scaled item scores ranged from 0.41 to 0.53 with standard deviations (across the five 
standards) the order of about 0.05. These results indicate that the difficulty levels of the five standards 
within a given assessment are similar to each other (small standard deviation) and are well aligned with 
the proficiency distributions of the students (means close to 0.50), neither overly difficult nor overly 
easy. In contrast to alternate assessments, the difficulty values for assessments designed for the 
general population tend to be in the 0.4 to 0.7 range for the majority of items. Because the nature of 
alternate assessments is different from that of general assessments, and because very few guidelines 
exist as to criteria for interpreting these values for alternate assessments, the values presented in 
Tables 9-2 and 9-3 should not be interpreted to mean that the students who took the NYSAA performed 
either better or worse than the students who took general assessments. 

Also shown in Tables 9-2 and 9-3 are the mean discrimination values for the S/LOC and 
Standards-based items, respectively, as calculated by the correlation between the item scores and the 
scaled total scores. Because the majority of students received high scores on the S/LOC items and 
these raw scores are not adjusted for LOC as is done for the scaled total scores, the discrimination 
indices are somewhat lower than one might otherwise expect, with mean values ranging from 0.27 to 
0.47. In particular, if all of the students receive high percent-correct scores on the S/LOC items, there is 
little variability in the item score for differentiating the criterion scores. On the other hand, for the 
Standards-based items, the mean discrimination indices ranged from 0.53 to 0.70 for mathematics and 


ELA and 0.45 to 0.58 for science and social studies. These results indicate a strong positive 
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relationship between the scaled scores on the Standards-based items and the scaled total scores. As 
with the item difficulty values, because the nature and use of the NYSAA are different from those of a 
general assessment, and because very few guidelines exist as to criteria for interpreting these values 
for alternate assessments, the statistics presented in Tables 9-2 and 9-3 should be interpreted with 


caution. 


Table 9-2. 2013-14 NYSAA: Summary of Item Difficulty and Discrimination Statistics 
by Content Area and Grade for S/LOC Items 


Number _ Difficulty (p-Value) Discrimination 

Content Area Grade oF iems Ween SD Mean SD 
3 15 0.86 0.05 0.39 0.16 

4 15 0.88 0.05 0.38 0.13 

English 5 15 0.86 0.05 0.44 0.08 
Language 6 15 0.87 0.04 0.40 0.13 
Arts 7 15 0.86 0.03 0.38 0.15 

8 15 0.87 0.05 0.41 0.10 

High School 15 0.84 0.04 0.39 0.15 

3 15 0.84 0.06 0.45 0.18 

4 15 0.85 0.04 0.41 0.12 

5 15 0.84 0.05 0.41 0.16 

Mathematics 6 15 0.85 0.05 0.41 0.19 
7 15 0.83 0.07 0.47 0.10 

8 15 0.85 0.05 0.44 0.13 

High School 15 0.81 0.04 0.42 0.11 

4 6 0.89 0.04 0.30 0.11 

Science 8 6 0.87 0.05 0.33 0.07 
High School 6 0.86 0.04 0.27 0.15 

Social Studies High School 6 0.84 0.07 0.32 0.16 


Table 9-3. 2013-14 NYSAA: Summary of Item Difficulty and Discrimination Statistics by Content Area and 
Grade for Standards-based Items 


Difficulty Discrimination 
Content Area Grade eae Scaled Score p-Value (Corr w/Total) 
lean Std Mean Std Mean Std 
3 5 11.88 1.71 0.48 0.07 0.59 0.05 
4 5 11.70 1.45 0.47 0.06 0.56 0.05 
English 5 5 10.96 1.12 0.44 0.04 0.53 0.05 
Language 6 5 11.63 0.98 0.47 0.04 0.61 0.02 
Arts 7 5 10.40 0.76 0.42 0.03 0.58 0.01 
8 5 10.20 0.93 0.41 0.04 0.60 0.04 
High School 5 11.82 0.94 0.47 0.04 0.61 0.05 
3 5 11.86 1.77 0.47 0.07 0.65 0.04 
4 5 11.20 1.22 0.45 0.05 0.60 0.05 
Mathematics 5 5 10.30 1.21 0.41 0.05 0.61 0.03 
6 5 10.93 0.86 0.44 0.03 0.70 0.02 
7 5 11.60 1.43 0.46 0.06 0.56 0.05 
continued 
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as Difficulty Discrimination 
Grade nad 


Content Area of ems Scaled Score p-Value (Corr w/Total) 
Mean Std Mean Std Mean Std 
Nothanranine . 8 5 10.21 1.38 0.41 0.06 0.62 0.04 
High School 5 12.16 1.04 0.49 0.04 0.54 0.08 
4 2 11.44 0.44 0.46 0.02 0.45 0.01 
Science 8 2 12.84 0.60 0.51 0.02 0.45 0.00 
High School 2 13.13 0.10 0.53 0.00 0.54 0.01 
Social Studies High School 2 13.27 0.03 0.53 0.00 0.58 0.02 


9.2 STRUCTURAL RELATIONSHIP 


By design, the performance level classification of the NYSAA is based on scaled scores 
associated with five standards for ELA and mathematics, and two standards for science and social 
studies. These different standards can be conceptualized as different construct dimensions of the 
assessment. As with any assessment, it is important that the dimensions composing the assessment 
be carefully examined. This was achieved by exploring the relationships between student scaled scores 
on the different dimensions with Pearson correlation coefficients. A very low correlation (near zero) 
would indicate that the dimensions are not related; a low negative correlation (approaching -1.00) would 
indicate that they are inversely related (i.e., that a student with a high score on one dimension had a 
low score on the other); and a high positive correlation (approaching 1.00) would indicate that the 
information provided by one dimension is similar to that provided by the other dimension. In addition, 
the correlation matrices for the standards were analyzed with factor analysis to determine the number 
of dimensions that are statistically significant for each assessment instrument analyzed. Because these 
assessments are unidimensionally scored, it is important to determine the degree to which 
unidimensionality accounts for the variability in the scores. 

The average correlations between the scaled scores on the standards by content area and 


grade are shown in Table 9-4. The detailed results for each pair of standards are given in Appendix I. 


Table 9-4. 2013-14 NYSAA: Average Correlations 


Average Correlation 
Coven aie Grade eanciaicn Standard Deviation 
3 0.47 0.06 
4 0.45 0.05 
Language 6 0.49 0.03 
Arts 7 0.47 0.02 
8 0.49 0.04 
High School 0.51 0.06 
; 3 0.53 0.05 
Mathematics 4 0.48 0.06 
continued 
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Average Correlation 


oa Grade Correlation Standard Deviation 
5 0.50 0.05 
6 0.60 0.02 
Mathematics 7 0.43 0.06 
8 0.51 0.05 
High School 0.41 0.08 
4 0.49 
Science 8 0.48 
High School 0.58 
Social Studies High School 0.62 


The inter-item correlations ranged from a low of .29 (high school math, Standards 913 and 915) 
to a high of 0.65 (Grade 6 math, Standards 606 and 608). The averages for a given grade and content 
area ranged from 0.41 to 0.62. These correlations indicate that the scores on the different standards 
within a grade and content area have a strong positive relationship with each other. Next, a factor 
analysis was conducted on the correlation matrix for each grade and content area, to determine the 
degree to which a unidimensional scale can account for the variance in the scaled total scores. For 
science and social studies, because they have only two scored dimensions, the strong positive 
correlations are alone strong evidence in support of their unidimensional scales. The factor analysis 
confirmed this by indicating that 74% to 81% of the variance in the correlations of the standards-based 
scaled scores is accounted for by a unidimensional scale. For ELA and mathematics, the factor 
analysis results indicate that a single scored dimension accounts for 53% to 69% of the variance in the 


correlations of the standards-based scaled scores. 


Table 9-5. 2013-14 NYSAA: Factor Analysis Table 


Percent of Variance Accounted for by Each Factor 


Grade Content Area 
1 2 3 4 5 
. English Language Arts 57.3% 13.4% 10.8% 9.9% 8.5% 
Mathematics 62.5% 11.6% 10.2% 8.4% 7.3% 
English Language Arts 55.9% 13.6% 11.2% 10.5% 8.9% 
4 Mathematics 58.8% 12.6% 11.6% 9.0% 8.0% 
Science 74.0% 26.0% 
5 English Language Arts 53.0% 14.0% 13.4% 10.4% 9.2% 
Mathematics 60.9% 12.7% 9.8% 8.9% 7.7% 
6 English Language Arts 59.5% 11.2% 10.6% 9.5% 9.1% 
Mathematics 68.5% 8.5% 8.4% 7.7% 6.9% 
7 English Language Arts 57.3% 11.7% 11.2% 10.5% 9.2% 
Mathematics 54.6% 13.9% 11.3% 10.3% 9.9% 
English Language Arts 59.7% 11.6% 10.8% 9.7% 8.2% 
8 Mathematics 61.7% 12.0% 9.6% 9.3% 7A% 
Science 74.0% 26.0% 
continued 
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Percent of Variance Accounted for by Each Factor 


Grade Content Area 


1 2 3 4 5 
English Language Arts 60.5% 12.3% 11.2% 9.0% 7.0% 
; Mathematics 53.5% 15.0% 12.3% 10.3% 9.0% 
High Schook_ ———_——_——_ 
Science 79.0% 21.0% 
Social Studies 81.0% 19.0% 


Thus, these results also give strong support for the appropriateness of the use of a 


unidimensional scale for these assessment instruments. 


9.3 BIAS/FAIRNESS 


The Code of Fair Testing Practices in Education (Joint Committee on Testing Practices, 2004) 
explicitly states that subgroup differences in performance should be examined when sample sizes 
permit, and actions should be taken to make certain that differences in performance are due to 
construct-relevant, rather than irrelevant, factors. The guidelines in the Code of Fair Testing Practices 
in Education are consistent with the relevant sections of the Standards for Educational and 
Psychological Testing (AERA, APA, NCME, 2014). 

When appropriate, the standardization differential item functioning (DIF) procedure (Dorans & 
Kulick, 1986) is used to identify items for which subgroups of interest perform differently, beyond the 
effect of differences in overall achievement. However, because the NYSAA uses a datafolio that does 
not include standard items that are taken by all students, it was not possible to conduct DIF analyses. 

Although it is not possible to run quantitative analyses of item bias for the NYSAA, due to data 
limitations, fairness can be addressed through the assessment Blueprints, which are designed to reflect 
the core curriculum, as described in detail earlier in this report. The development of the assessment 
Blueprints, which reflect recommendations laid out in the Standards for Educational and Psychological 
Testing, were designed to ensure that the test is free of any insensitive or offensive material, as well as 
to ensure alignment with general education grade-level content and to promote higher expectations for 
students taking the NYSAA. 

Issues of fairness are also addressed in the NYSAA Administration and Scoring Procedures. 
Chapter 7 of this report describes in detail the procedures for administering the NYSAA and 
constructing the datafolio, as well as the training and review steps designed to ensure that the test is 
administered appropriately and consistently for all students. Chapter 8 describes in detail the Scoring 
Rubrics used, selection and training of Scorers, and scoring quality control procedures. These 
processes were followed in order to minimize bias due to differences in how individual Scorers award 


scores. 
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CHAPTER 10 CHARACTERIZING ERRORS ASSOCIATED WITH 
TEST SCORES 


One of the primary uses of the New York State Alternate Assessment (NYSAA) scores is for 
school-, district-, and state-level accountability in the federal No Child Left Behind (NCLB) Act and in 
state accountability systems. The students are classified as Proficient or Not Proficient, and are 
included in the State’s Adequate Yearly Progress (AYP) calculation. In this case, the reliability of 
individual student scores, while not meaningless, becomes much less important. The scores have been 
collapsed for each student to a yes/no decision and then aggregated across students. 

For purposes of calculating reliability estimates, Standards-based item scores are defined in the 
same way as described in Chapter 9. Specifically, the scaled scores on the five standards for English 
Language Arts (ELA) and mathematics and the two standards for science and social studies are treated 


as the item scores. 


10.1 RELIABILITY 


In the previous chapter, individual item characteristics of the 2013-14 NYSAA were presented. 
Although individual item performance is an important focus for evaluation, a complete evaluation of an 
assessment must address the way in which the items (or, in this case, standards-based items) that 
make up the test score function together and complement one another. Any measurement includes 
some amount of measurement error. No academic assessment can measure student performance with 
perfect accuracy; some students will receive scores that underestimate their true abilities, and other 
students will receive scores that overestimate their true abilities. Items that function well together 
produce assessments that have less measurement error (i.e., the error is small, on average). Such 
assessments are described as reliable. 

There are a number of ways to estimate an assessment’s reliability. One approach is to split all 
test items into two groups and then correlate students’ scores on the two half-tests. This is known as a 
split-half estimate of reliability. If the two half-test scores correlate highly, the items on them are likely 
measuring very similar knowledge or skills. It suggests that measurement error will be minimal. 

The split-half method requires psychometricians to select items that contribute to each half-test 
score. This decision may have an effect on the resulting correlation, since each different possible split 
of the test halves will result in a different correlation. Another problem with the split-half method of 
calculating reliability is that it underestimates reliability, because test length is cut in half. All else being 
equal, a shorter test is less reliable than a longer test. Cronbach (1951) provided a statistic, alpha (a), 
that avoids the shortcomings of the split-half method by comparing individual item variances to total test 
variance. Cronbach's awas used to assess the reliability of the 2013-14 NYSAA tests. The formula is 


as follows: 


2013-14 NYSAA Technical Report: Chapter 10—Characterizing Errors Associated with Test Scores - 52 - 


where 
Zindexes the item, 
nis the number of items, 


2 : £03 : i 
O(y,) fepresents individual item variance, and 
L, 


a2 represents the total test variance. 


If the correlation is high (in practice, toward the high end of the typical Cronbach's a range of 
0.50 to 0.99), the parts of the test are likely measuring very similar knowledge or skills. Thus, a high 
Cronbach’s a coefficient is evidence that the standards-based items complement one another and 
suggests that the assessment is reliable. Table 10-1 presents scaled total score descriptive statistics 
(maximum possible scaled total score, average scaled total score, and standard deviation), Cronbach's 
a coefficient, and scaled total score standard errors of measurements (SEMs) for each content area 
and grade. The results show the reliability estimates range from 0.76 to 0.88 for ELA and mathematics, 
while ranging from 0.65 to 0.76 for science and social studies. The latter values are expected to be 
lower because those tests have fewer items. Considering that the NYSAAs are necessarily shorter than 
general assessments, the reliability coefficients in Table 10-1 give strong support to the reliability of the 


reported scaled total scores and their intended interpretations. 


Table 10-1. 2013-14 NYSAA: Scaled Total Score Descriptive Statistics, Cronbach’s Alpha, and 
Standard Errors of Measurement (SEM) by Grade and Content Area 


NuimbORGR Scaled Total Score 


Content Area Grade Students “uesinuin’. Jaan Standard Reliability (a) SEM 


Deviation 
3 2,687 525 458.07 20.89 0.81 9.13 
4 3,048 525 456.48 20.61 0.79 9.35 
English 5 3,122 525 452.88 18.57 0.76 9.01 
Language 6 3,298 525 456.64 20.11 0.83 8.40 
Arts 7 3,365 525 449.91 19.82 0.81 8.71 
8 3,167 525 449.24 18.62 0.82 7.79 
High School 3,050 525 457.00 20.89 0.84 8.46 
3 2,691 525 457.92 21.35 0.85 8.38 
4 3,046 525 454.56 22.07 0.81 9.69 
5 3,113 525 449.67 19.94 0.83 8.25 
Mathematics 6 3,299 525 453.07 21.60 0.88 7.49 
7 3,362 525 455.82 22.24 0.79 10.22 
8 3,162 525 449.46 19.84 0.83 8.14 
High School 3,045 525 458.63 22.87 0.78 10.68 
4 3,025 600 572.16 9.07 0.65 5.33 
Science 8 3,150 600 574.75 10.04 0.65 5.93 
High School 3,031 600 575.50 10.35 0.73 5.37 
Social Studies High School 3,031 600 575.88 9.81 0.76 4.80 
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10.2 SUBGROUP RELIABILITY 


The reliability coefficients discussed in the previous section were based on the overall 
population of students who took the 2013-14 NYSAA. Subgroup Cronbach's a’s were calculated using 
the formula defined above, using only the members of the subgroup in question in the computations. 
These statistics are reported in Appendix D. Note that statistics are only reported for subgroups with at 
least 11 students. 

For several reasons, the results of this section should be interpreted with caution. First, inherent 
differences between grades and content areas preclude making valid inferences about the quality of a 
test based on statistical comparisons with other tests. Second, reliabilities are dependent not only on 
the measurement properties of a test but on the statistical distribution of the studied subgroup. For 
example, it can be readily seen in Appendix D that subgroup sample sizes may vary considerably, 
which results in natural variation in reliability coefficients. Alternatively, a, which is a type of correlation 
coefficient, may be artificially depressed for subgroups with little variability (Draper & Smith, 1998). 
Third, there is no industry standard to interpret the strength of a reliability coefficient, and this is 


particularly true when the population of interest is a single subgroup. 


10.3 DECISION ACCURACY AND CONSISTENCY 


While related to reliability, the accuracy and consistency of classifying students into 
performance categories is an even more important issue in a standards-based reporting framework 
(Livingston & Lewis, 1995). Unlike generalizability coefficients, decision accuracy and consistency 
(DAC) can usually be computed with the data currently available for most alternate assessments. 
Based on the raw scale cut scores established for each content area via standard setting in June 2008, 
each student was classified into one of the following performance levels: Not Meeting Learning 
Standards, Partially Meeting Learning Standards, Meeting Learning Standards, and Meeting Learning 
Standards with Distinction. (Lookup tables for converting raw scores to performance levels are 
presented in Chapter 11.) This section of the report explains the methodologies used to assess the 
reliability of classification decisions and presents the results. 

Accuracy refers to the extent to which decisions, based on test scores match decisions that 
would have been made if the scores did not contain any measurement error. Accuracy must be 
estimated, because errorless test scores do not exist. Consistency measures the extent to which 
classification decisions based on test scores match the decisions based on scores from a second, 
parallel form of the same test. Consistency can be evaluated directly from actual responses to test 
items if two complete and parallel forms of the test are given to the same group of students. In 
operational test programs, however, such a design is usually impractical. Instead, techniques have 


been developed to estimate both the accuracy and the consistency of classification decisions based on 
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a single administration of a test. The Livingston and Lewis (1995) technique was used for the NYSAA 
because it is easily adaptable to all types of testing formats. 

The accuracy and consistency estimates reported in the tables below make use of “true scores” 
in the classical test theory sense. A true score is the score that would be obtained if a test had no 
measurement error. Of course, true scores cannot be observed and, therefore, must be estimated. In 
the Livingston and Lewis method, estimated true scores are used to categorize students into their “true” 
classifications. 

For the NYSAA, after various technical adjustments (described in Livingston and Lewis, 1995), 
a four-by-four contingency table of accuracy was created for each content area and grade, where cell [/, 
j\ represented the estimated proportion of students whose true score fell into classification 7(where /= 1 
to 4) and observed score into classification 7 (where /= 1 to 4). The sum of the diagonal entries (i.e., the 
proportion of students whose true and observed classifications matched) signified overall accuracy. 

To calculate consistency, true scores were used to estimate the joint distribution of 
classifications on two independent, parallel test forms. Following statistical adjustments per Livingston 
and Lewis (1995), a new four-by-four contingency table was created for each content area and grade 
and populated by the proportion of students who would be categorized into each combination of 
classifications according to the two (hypothetical) parallel test forms. Cell [7, 7] of this table represented 
the estimated proportion of students whose observed score on the first form would fall into classification 
i (where 7/= 1 to 4) and whose observed score on the second form would fall into classification 7 (where 
j= 1 to 4). The sum of the diagonal entries (i.e., the proportion of students categorized by the two forms 
into exactly the same classification) signified overall consistency. 

Another way to measure consistency is to use Cohen’s (1960) coefficient « (kappa), which 
assesses the proportion of consistent classifications after removing the proportion of consistent 


classifications that would be expected by chance. It is calculated using the following formula: 


__ (Observed agreement)—(Chance agreement) — YiCij—LiCi.Ci 
_ 1-(Chance agreement) 7 1-YiCi.Ci 


where 
C, is the proportion of students whose observed performance level would be Level / (where 7= 1 — 4) on the 
first hypothetical parallel form of the test, 


C, is the proportion of students whose observed performance level would be Level 7 (where /= 1 — 4) on the 
second hypothetical parallel form of the test, and 


C;; is the proportion of students whose observed performance level would be Level / (where /= 1 — 4) on 
both hypothetical parallel forms of the test. 
Because x is corrected for chance, its values are lower than those of other consistency estimates. 
The accuracy and consistency analyses described above are provided in Table 10-2. The table 
includes overall accuracy and consistency indices, including kappa. Accuracy and consistency values 


conditional upon performance level are also given. For these calculations, the denominator is the 
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proportion of students associated with a given performance level. For example, the conditional 
accuracy value is 0.84 for Meeting Learning Standards for Grade 3 ELA. This figure indicates that 
among the students whose true scores placed them in this classification, 84% would be expected to be 
in this classification when categorized according to their observed scores. Similarly, a consistency 
value of 0.80 indicates that 80% of students with observed scores in the Not Meeting Learning 
Standards level would be expected to score in this classification again if a second, parallel test form 


were used. 


Table 10-2. 2013-14 NYSAA: Summary of Decision Accuracy (and Consistency) Results 

by Content Area and Grade—Overall and Conditional on Performance Level 

Conditional on Level 

Content Area Grade Overall Kappa Not Partially 
Meeting Meeting 
3 0.79 (0.71) 0.47 0.68 (0.48) 0.68(0.59) 0.84(0.80) 0.79 (0.54) 
4 0.80 (0.72) 0.45 0.64(0.39) 0.69(0.59) 0.85(0.81) 0.79 (0.52) 
English 5 0.78 (0.69) 0.41 0.65(0.41) 0.65(0.54) 0.84(0.80) 0.77 (0.48) 
6 
7 


Meeting with 


Meeting Distinction 


Language 0.81 (0.74) 0.50 0.69(0.50) 0.70(0.61) 0.86 (0.82) 0.79 (0.54) 
Arts 0.76 (0.68) 0.44 0.73(0.58) 0.54(0.44) 0.85(0.80) 0.78 (0.50) 
8 0.75 (0.68) 0.45 0.74(0.61) 0.49(0.38) 0.84(0.80) 0.79 (0.55) 

High School 0.79(0.71) 0.50 0.71(0.54) 0.69(0.60) 0.85(0.80) 0.80 (0.57) 

3 0.79(0.71) 0.50 0.75(0.61) 0.63(0.53) 0.86 (0.82) 0.81 (0.57) 

4 0.76 (0.67) 0.44 0.71(0.55) 0.57(0.46) 0.83(0.78) 0.80 (0.56) 

5 0.80 (0.72) 0.48 0.71(0.53) 0.65(0.55) 0.86 (0.82) 0.79 (0.54) 

Mathematics 6 0.81 (0.74) 0.57 0.74(0.58) 0.75(0.67) 0.86 (0.81) 0.83 (0.65) 
i 0.76 (0.68) 0.43 0.67(0.47) 0.63(0.53) 0.83(0.78) 0.79 (0.51) 

8 0.77 (0.69) 0.47 0.73(0.58) 0.63(0.53) 0.85 (0.80) 0.79 (0.55) 

High School 0.78(0.70) 0.41 0.66 (0.45) 0.60(0.49) 0.85(0.81) 0.79 (0.49) 

4 0.76 (0.67) 0.30 0.58(0.31) 0.52(0.40) 0.83(0.80) 0.75 (0.36) 

Science 8 0.74 (0.65) 0.30 0.56(0.28) 0.54(0.43) 0.81(0.77) 0.78 (0.40) 


High School 0.76 (0.68) 0.35 0.59(0.33) 0.58(0.46) 0.82(0.79) 0.79 (0.46) 
Social Studies High School 0.74 (0.64) 0.41 0.61(0.36) 0.64(0.53) 0.75(0.70) 0.83 (0.63) 


For some testing situations, of greatest concern may be decisions around level thresholds. For 
example, in testing done for No Child Left Behind (NCLB) Act of 2001 accountability purposes, the 
primary concern is distinguishing between students who are proficient and those who are not yet 
proficient. In this case, the accuracy of the Partially Meeting/Meeting threshold is of greatest interest. 
Table 10-3 provides accuracy and consistency estimates at each cutpoint, as well as false positive and 
false negative decision rates. (A false positive is the proportion of students whose observed scores 
were above the cut and whose true scores were below the cut. A false negative is the proportion of 
students whose observed scores were below the cut and whose true scores were above the cut.) 

The indices described above are derived from Livingston and Lewis’s (1995) method of 


estimating the accuracy and consistency of classifications. It should be noted that Livingston and Lewis 
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discuss two versions of the accuracy and consistency tables. A standard version performs calculations 
for forms parallel to the form taken. An “adjusted” version adjusts the results of one form to match the 
observed score distribution obtained in the data. The tables on the previous pages use the standard 
version for two reasons: (1) this “unadjusted” version can be considered a smoothing of the data, 
thereby decreasing the variability of the results; and (2) for results dealing with the consistency of two 
parallel forms, the unadjusted tables are symmetrical, indicating that the two parallel forms have the 
same statistical properties. This second reason is consistent with the notion of forms that are parallel; 
that is, it is more intuitive and interpretable for two parallel forms to have the same statistical 
distribution. 

Note that, as with other methods of evaluating reliability, DAC statistics calculated based on 
small groups can be expected to be lower than those calculated based on larger groups. For this 
reason, the values presented in the tables below should be interpreted with caution. In addition, it is 
important to remember that it is inappropriate to compare DAC statistics between grades and content 


areas. 
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Table 10-3. Summary of Decision (and Consistency) Results 
by Content Area and Grade—Conditional on Cutpoint 


Not Meeting / Partially Meeting / Meeting / 
Partially Meeting Meeting Meeting with Distinction 
Content Area Grade Accuracy False Accuracy False Accuracy False 
(consistency) Positive Negative (consistency) Positive Negative (consistency) Positive Negative 

3 0.96 (0.95) 0.01 0.02 0.87 (0.83) (0.07 0.06 0.95 (0.93) 0.04 0.01 
4 0.98 (0.97) 0.01 0.02 0.88 (0.83) 0.06 0.06 0.95 (0.93) 0.04 0.01 
English 5 0.97 (0.95) 0.01 0.02 0.86 (0.81) 0.07 0.07 0.95 (0.93) 0.04 0.01 
Language 6 0.97 (0.95) 0.01 0.02 0.88 (0.84) 0.06 0.06 0.96 (0.94) 0.03 0.01 
Arts 7 0.92 (0.89) 0.03 0.05 0.87 (0.81) 0.07 0.06 0.97 (0.95) 0.03 0.01 
8 0.92 (0.88) 0.04 0.05 0.87 (0.82) (0.07 0.06 0.96 (0.94) 0.03 0.01 
High School 0.96 (0.94) —- 0.02 0.03 0.88 (0.83) 0.06 0.06 0.95 (0.94) 0.04 0.01 
3 0.94 (0.92) 0.02 0.03 0.88 (0.84) 0.06 0.05 0.96 (0.94) 0.03 0.01 
4 0.94 (0.91) 0.03 0.04 0.87 (0.82) 0.07 0.06 0.94 (0.92) 0.04 0.01 
5 0.95 (0.93) 0.02 0.03 0.88 (0.83) 0.06 0.06 0.96 (0.95) 0.03 0.01 
Mathematics 6 0.96 (0.95) 0.01 0.02 0.90 (0.86) 0.06 0.05 0.96 (0.94) 0.03 0.01 
7 0.95 (0.93) 0.02 0.03 0.86 (0.81) 0.07 0.06 0.95 (0.93) 0.04 0.01 
8 0.94 (0.91) 0.03 0.04 0.87 (0.82) (0.07 0.06 0.96 (0.94) 0.03 0.01 
High School 0.96 (0.94) 0.01 0.03 0.87 (0.82) 0.07 0.06 0.95 (0.93) 0.04 0.01 
4 0.96 (0.94) 0.01 0.03 0.85 (0.79) 0.08 0.08 0.95 (0.93) 0.05 0.00 
Science 8 0.96 (0.95) 0.01 0.03 0.85 (0.79) 0.08 0.08 0.93 (0.90) 0.07 0.00 
High School 0.97 (0.95) 0.01 0.02 0.87 (0.82) (0.07 0.06 0.92 (0.89) 0.07 0.01 
Social Studies High School 0.98 (0.96) 0.01 0.02 0.88 (0.83) 0.06 0.06 0.88 (0.84) 0.09 0.03 
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10.4 INTERRATER CONSISTENCY 


Chapter 9 of this report describes in detail the processes that were implemented to monitor the 
quality of the hand-scoring of student responses for polytomous items. One of these processes was 
double-blind scoring of all student responses. Results of the double-blind scoring were used during 
scoring to identify Scorers who required retraining or other intervention, and are presented here as 
evidence of the reliability of the NYSAA. A summary of the interrater consistency results is presented in 
Table 10-4. Results in the table are collapsed across the tasks by content area and grade. The table 
shows the number of included scores, the percent exact agreement, the correlation between the first 
two sets of scores, the mean absolute difference between scores that did not have exact agreement, 
and the standard deviation of these absolute differences. This same information is provided at the item 


level in Appendix E. 


Table 10-4. Summary of Interrater Consistency Statistics 
Collapsed across Items by Content Area and Grade 


Overall Overall Overall Mean Overall 
: Interrater Interrater Absolute S.D. 
Subject Grade y Percent Exact Correlation Difference for Absolute 
Agreement Agreement Non-Exact Difference 
3 3,150 98.38 1.00 2.53 2.06 
4 3,656 98.52 1.00 2.76 2.81 
English 5 3,735 98.26 0.99 3.04 3.01 
Language 6 4,078 98.23 0.99 2.87 3.06 
Arts 7 3,778 98.44 1.00 2.05 2.69 
8 3,657 98.28 1.00 2.34 2.62 
High School 3,148 97.87 1.00 2.42 2.77 
3 3,108 98.26 1.00 2.70 2.47 
4 3,719 97.98 1.00 2.63 2.61 
5 3,725 97.61 0.99 2.72 2.89 
Mathematics 6 4,088 97.92 0.99 2.94 3.02 
7 3,777 97.48 1.00 2.54 2.86 
8 3,694 98.11 1.00 2.40 2.73 
High School 3,193 97.75 1.00 2.95 2.63 
4 1,494 98.39 0.99 2.98 2.76 
Science 8 1,466 98.57 1.00 2.34 3.01 
High School 1,296 98.38 1.00 2.50 2.25 
Social Studies High School 1,308 98.32 0.99 3.03 3.15 
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CHAPTER 11 COMPARABILITY (SCALING AND EQUATING) 


11.1 COMPARABILITY OF SCORES ACROSS YEARS 


In administering the New York State Alternate Assessment (NYSAA), teachers select 
Extensions or Alternate Grade-Level Indicators (AGLIs), following the Test Blueprints. Use of the 
Extensions or AGLIs and Blueprints ensures that the assessment, as it is administered, is appropriate 
for the individual needs of the student being assessed and that the standards required are covered. 
The process enables teachers to customize the assessment for individual students while, at the same 
time, ensuring comparability across years through the use of the same Blueprints, Extensions/AGLIs, 
and Scoring Rubrics from year to year. Additionally, comparability is ensured through the scoring 
process. Teachers use the same Scoring Rubric for a datafolio each year, and scoring occurs at 
regional Scoring Institutes that all follow the same Scoring Training program and Scoring Procedures, 
as well as the standard scoring quality control processes, as described in Chapter 8. Additional 
processes to ensure across-year comparability include calculation of reported scores and 


categorization into achievement levels, as described below. 


11.1.1 Standard Setting 


Standard setting was conducted in June 2014 to establish cut scores for the scaled total scores 
for each alternate performance level in English Language Arts (ELA) and mathematics, Grades 3 
through 8 and high school; in science, Grades 4, 8, and high school; and in high school social studies. 
To ensure continuity of score reporting across years, the cuts that were established at the standard- 
setting meeting will continue to be used in future years, until it is necessary to reset standards. The 
scaled total score cutpoints for the NYSAA as established via standard setting are presented in Table 
11-1. 


Table 11-1. 2013-14 NYSAA: Cut Scores on the Theta Metric and Reporting Scale 
by Content Area and Grade 


ea tenF Area Coe Raw Score Cuts Raw Score 
Cut1 Cut2 Cut3 Minimum Maximum 
3 426 449 491 400 525 
English 4 421 445 489 400 525 
Language 5 424 444 484 400 525 
Arts 6 425 447 490 400 525 
7 429 443 486 400 525 
continued 
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Raw Score Cuts Raw Score 


CONTENTS nigee Cut1 Cut2 Cut3 Minimum Maximum 

English 8 431 442 480 400 525 
Language . 

Ate High School 427 449 490 400 525 
3 432 450 493 400 525 
4 428 445 488 400 525 
5 423, 441 485 400 525 
Mathematics 6 422 445 485 400 525 
‘t 425 447 492 400 525 
8 426 443 483 400 525 
High School 425 446 496 400 525 
4 559 567 589 550 600 
Science 8 559 569 591 550 600 
High School 559 569 591 550 600 
Social Studies High School 559 570 586 550 600 


Table F-1 in Appendix F shows performance level distributions for 2014 by content area and grade. 


11.1.2 Reported Scores (Cumulative Distributions) 


Students’ entry scores are calculated based on their Level of Accuracy and Level of Complexity 
scores for each of the final date of student performance of the Extensions or AGLIs in a given entry. 
The overall score is then the sum of the entry scores. Because of the use of the formula, there may be 
multiple ways that a student can attain a given total score. 

Graphs of the cumulative reported raw score distributions for 2014 are provided in Appendix G. 


As the curves move to the right, they represent an increase in performance. 


11.1.3 Performance Level Distributions 


Appendix F shows the percentages of students earning scores at each performance level. A 
score of No Score (NS) is designated if a datafolio does not adhere to the administration guidelines. 
(Complete information regarding scoring can be found in the two scoring documents entitled Steps for 
Scoring 2013-14 NYSAA Datafolios and Decision Rules for Scoring 2013-14 NYSAA Datafolios.) The 


percentages are presented by grade, content area, and Performance Level. 


11.2 LINKAGES ACROSS GRADES 
In developing the NYSAA, a content-based approach for addressing continuity across grades 


was implemented. Specifically, issues of continuity were addressed in the following processes: (1) 
development, (2) administration, and (3) standard setting. 
As explained in Chapter 4, the Extensions and AGLIs describe the content to be included in 


students’ instructional programs for each grade level. The Extensions in ELA and mathematics are 
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based on the Common Core Learning Standards (CCLS) and AGLIs in science and social studies are 
based on the core curriculum’s Grade-Level Expectations, but have been reduced in depth and 
breadth. The Extensions and AGLIs are designed to follow a developmental continuum of skills that 
increases across grades. Each Assessment Task must align to the Extension or AGLI, and are 
designed to measure grade-specific content and skills. These Assessment Tasks and the Extensions or 
AGLIs, along with Test Blueprints, were designed to mirror the developmental continuum reflected in 
the Extensions and AGLIs and to ensure that each datafolio builds upon the appropriate Knowledge and 
skills, thereby reflecting the desired continuity across grades. 

During administration, the Test Blueprint serves as a guide to the teachers as to how to select 
Extensions and AGLIs that are appropriate for a given student. In addition, teachers must select 
Assessment Tasks that are aligned with the Extensions and AGLIs chosen. As with other aspects of the 
development and administration of the NYSAA, use of the Test Blueprints and the Extensions and 
AGLIs ensures that the student is being assessed at a level that is appropriate for his or her individual 
needs and that the Extensions or AGLIs and Assessment Tasks to which students are exposed follow a 
developmentally appropriate continuum from year to year. Thus, linkages across grades are built into 
the design of the datafolio. 

Finally, the continuity of the NYSAA across grades was further verified through the standard- 
setting procedures. The achievement level descriptors used for standard setting were based on the 
student expectations as delineated in the Extensions and AGLIs. Proficiency across grades, therefore, 
was expected to follow the developmental continuum established by the Extensions or AGLIs and, thus, 


to reflect a higher level of cognition as the grades increased. 
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CHAPTER 12 VALIDITY 


12.1 PROCEDURAL VALIDITY 


In order to ensure consistency of the information given to teachers across New York State, sets 


of documents and training programs were developed and distributed statewide. New York State has a 


set of Alternate Assessment Training Network Specialists (AATN Specialists) and Score Site 


Coordinators (SSCs) who present a turnkey training provided to them by the New York State Education 


Department (the Department) and Measured Progress. 


For the administration of the New York State Alternate Assessment (NYSAA), the materials 


included the following: 


NYSAA Administration Manual: This document contains all the guidelines and specific 
requirements of the NYSAA; all the forms required to be used in the datafolio; and the 
Test Blueprints, Extensions for English Language Arts (ELA) and mathematics, Alternate 
Grade-Level Indicators (AGLIs) for science and social studies, and Assessment Tasks 
for each Extension or AGLI for each grade level and content area. 


Training program DVD: The entire Administration Training program that is used with 
teachers is contained in this recorded program. All AATN Specialists are required to use 
the DVD in its entirety to train teachers. It ensures that the exact same message Is 
imparted statewide. 


Training program slides and handouts: All slides and handouts developed by the 
Department and Measured Progress are required to be used by the AATN Specialists 
while training teachers. The handouts include slide printouts and Guided Practice 
activities. 


For the scoring of the NYSAA, the materials included the following: 


Steps for Scoring 2013-14 NYSAA Datafolios and Decision Rules for Scoring 2013-14 
NYSAA Datafolios: These are the two main documents used to guide the scoring 
process for each datafolio (see Appendices B and C). 


Training program DVD: The entire Scoring Training program that is used with Scorers is 
contained in this recorded program. All SSCs and AATN Specialists are required to use 
the DVD in its entirety to train Scorers. It ensures that the exact same message is 
imparted statewide. 


Datafolio practices and qualifiers: All Scorers must complete the three practice samples 
provided and then must qualify by scoring datafolio samples. All Scorers are qualified 
using calibrated materials that were initially identified during a Benchmarking process. 
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12.2 CONTENT VALIDITY 


The Standards for Educational and Psychological Testing (AERA, APA, NCME, 2014) notes 
that an important part of establishing test validity is ensuring that a close, substantive relationship exists 
between a test’s content and the underlying construct it is intended to measure. The Standards further 
elaborate that the test content refers to the “themes, wording, and format of the items, tasks, or 
questions on a test. Administration and scoring may also be relevant to content-based evidence” (2014, 
p. 14). In addition to describing the content in detail, content validity evidence must, of course, relate 
the content to the construct the test is intended to measure. One important approach in this regard 
mentioned in the Standards is the use of “expert judgment of the relationship between parts of the test 
and the construct” (2014, p. 14). 

The New York State (NYS) learning standards provide the framework for the New York State 
Testing Program, including the NYSAA. For ELA and mathematics, the standards are from the NYS 
Common Core Learning Standards (CCLS). For science and social studies, the standards are from the 
NYS learning standards. These standards are the constructs that are intended to be measured by the 
NYSAA. Chapters 4 through 6 of this report describe in detail the development and design of the 
content for the NYSAA, with special emphasis on the relationship of the test content to the standards. 
Chapter 8 provides a detailed description of the scoring process for the NYSAA, again emphasizing 
that the procedures used ensure strong adherence to the standards. Another important component of 
the Scoring Procedure is the standard-setting process, in which expert judgment is used to set the 
scores on the test that correspond to different levels of classification of student achievement relative to 
the standards. The Standard Setting Report documenting the June 2014 standard-setting meeting 
describes the rigorous procedures that were adhered to in order to ensure that the content-related 
aspects of the standard-setting maintained a strong substantive alignment with the standards. 

As shown from the above definition of construct validity and in the descriptions of the contents 
of Chapters 4, 5, 6, and 8 of this report, a complete description of the content validity of the NYSAA is 


available to the reader. 


12.3. CONSEQUENTIAL VALIDITY 


Beginning in 1997, the Department began discussions on how to provide students who have 
severe cognitive disabilities access to the general education standards. To that end, an advisory 
committee made up of New York State Special education and general education teacher committees 
was formed. Their goal was to develop a handbook that would provide teachers with an alternate 
pathway for this group of students to gain access to the NYS learning standards. On July 17, 1997, the 


New York State Board of Regents endorsed a set of Alternate Performance Indicators (APIs) that were 
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linked to the NYS learning standards. The purpose of the APIs was to provide teachers with a way of 
teaching academic content to students with severe cognitive disabilities. The final manual, entitled The 
Learning Standards and Alternate Performance Indicators for Students with Severe Disabilities, was 
published in 1998 and distributed statewide. 

As mandated in the reauthorized Individuals with Disabilities Education Act of 1997 (IDEA of 
1997), states were required to have an alternate assessment in place by July 2000 for those students 
who could not participate in the general education assessment, even with accommodations. Because of 
the groundbreaking work already done, the Department, in collaboration with Measured Progress and 
under the guidance of the advisory committee, endorsed the use of the APIs in 1997 as a way to 
measure the knowledge, skills, and understanding of students with severe cognitive disabilities against 
the NYS learning standards. The advisory committee concluded that all students must be given the 
opportunity to achieve the learning standards, but that not all standards are appropriate for this group of 
students, which was in line with the intent of the IDEA of 1997. It was understood that this group of 
students would be assessed against APIs because of their inability to participate in the general 
assessment, even with accommodations. The APIs, while based on the learning standards, were, by 
their very nature, functional and limited to students with severe cognitive disabilities. They reflected 
what was determined to be appropriate for this group of students. They were not grade specific, nor 
were they aligned to grade-level content. The Committees on Special Education (CSEs) determined 
which students were appropriate for the NYSAA, based on several strict criteria, and decided which 
APIs the students would be assessed on. The first NYSAA was piloted between March 1998 and March 
2000, with full implementation during the 2000-01 school year. The purpose of the NYSAA was to 
promote the inclusion of students with severe cognitive disabilities in the statewide assessment 
program. It was not for the purposes of Adequate Yearly Progress (AYP) as defined by the No Child 
Left Behind (NCLB) Act of 2001. 

The following is the calendar of events the Department followed to develop and implement its 


first alternate assessment. 


Spring 1998 Conduct regional training for teachers on the APIs 

March 1998—March 2000 Develop and pilot the alternate assessment system 

March—June 2000 Provide information and training on the alternate assessment 
system 

July 2000 Implement a statewide alternate assessment system as required by 
IDEA of 1997 

June 2001 Collect data and report student scores to the public 


The Department and its Special education and general education teacher committees were 


committed to building an assessment and accountability system that included students with severe 
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cognitive disabilities. New York State was one of the first states to engage teachers, administrators, 
policymakers, and others in these important discussions, and it did pioneering work in the early years of 
alternate assessment. 

With the reauthorization of the NCLB Act of 2001, students in every state are being held to high 
levels of student academic achievement, including students with severe cognitive disabilities. The 
original NYSAA tested students in Grades 4, 8, and in high school in the content areas of ELA, 
mathematics, science/health, and social studies. Based on new testing of grade requirements in NCLB, 
in September 2005 the Department began to implement a revised NYSAA that included Grades 3 
through 8 and high school in the content areas of ELA, mathematics, science, and social studies. The 
students were assessed against the original APIs; however, the format and the number of APIs 


assessed were modified. Table 12-1 outlines the revised NYSAA. 


Table 12-1. NYSAA prior to 2013-14: Revised NYSAA 
Grades 3-8 and High School 


Grade Equivalents 


Datafolio Component 


Anchor Expanded 


4, 8, and High School 


3, 5, 6, and 7 


Table of contents 
Student Page 


One Entry Cover Sheet for each 
content area 


One Data Summary Sheet for each 


content area 


Verifying evidence per API 


Permission to tape and photograph 


Digital Video and Audio Clip 
Summary form 


v 
v 


English Language Arts, 
mathematics, social studies, 
science 


4 (one for each content area 
above) 


1 piece per API in each content 
area 


If applicable 
If applicable 


v 
v 


English Language Arts, 
mathematics 


2 (one for English Language 
Arts, one for mathematics) 


3 pieces for mandatory API in 
English Language Arts and 
mathematics 


If applicable 
If applicable 


During the 2005-06 testing cycle, the Department submitted its accountability documentation for 
peer review to the United States Education Department. The results of that review required the 


Department to revise its alternate assessment to ensure: 


» the presence of evidence of alignment between the NYSAA alternate achievement 
standards and the newly adopted Grade-Level Expectations; 


« that students are assessed at each required grade; 


« the setting of cutpoints and the development of Alternate Performance Level Descriptors 
(APLDs) for each grade level and content area; and 
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» the technical quality of the assessment, including research-based standard setting, and 
the production and submission of the Standard Setting Report and Technical Report. 


The new assessment system had to be in place for the 2006-07 testing cycle, culminating with 
standard setting in June 2007. 

Beginning in July 2006, the Department, in collaboration with Measured Progress, redesigned 
the NYSAA. The focus and purpose of the assessment is to ensure that students with severe cognitive 
disabilities are being provided access to the general education curriculum (e.g., Grade-Level 
Expectations). However, for these students, Grade-Level Expectations need to be expanded in both 
breadth and depth. This resulted in development of the AGLIs, which were contained in the NYSAA 
Administration Manual: Appendix H—NYSAA Frameworks. 

The Department brought together groups of Special education and general education teacher 
committees, including general education content specialists and Special Education Teachers, to 
develop the AGLIs. The groups referred to the general education Test Blueprints to determine the 
academic core priorities. From there, each content group reviewed the Grade-Level Expectations for 
each grade level and content area. The groups determined the Essences of the Grade-Level 
Expectations. Lastly, the groups wrote AGLIs that were aligned to the Essences of the Grade-Level 
Expectations. In addition to developing the AGLIs, Special education and general education teacher 
committees were also brought together to develop Sample Assessment Tasks (SATs) aligned to the 
AGLIs. The following year, the Stakeholder groups were brought in again to further refine what was 
originally developed. 

The new NYSAA was first implemented in the late fall of 2006. The administration, which had an 
abbreviated administration period, culminated with regional Scoring Institutes. Standard setting was 
conducted in June 2007, resulting in cut scores for each grade level and content area, as well as in 
APLDs. The cut scores were approved by the Commissioner of Education and submitted, along with 
the Standard Setting Report, to the United States Education Department. The 2007-08 NYSAA 
implementation occurred with a full administration period. This administration was based on the refined 
AGLIs and SATs. The administration again culminated with the regional Scoring Institutes. Standard 
setting was conducted on the revised AGLIs in June 2008, resulting in new cut scores and updated 
APLDs for each grade level and content area. The Commissioner of Education approved the updated 
cut scores in June 2008. The intent of the AGLIs was not changed following the 2007—08 
administration; therefore, the cut scores established during the June 2008 standard setting remained 
consistent for each grade level and content area. 

The New York State Board of Regents committed to the Common Core State Standards 
(CCSS) in January 2010 and formally adopted the CCSS for ELA and mathematics in July 2010 and 
incorporated New York State’s specific additions, creating the New York State P-12 CCLS in January 


2011. The Board of Regents announced that, for students with severe cognitive disabilities, student 
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progress on the CCLS would be measured beginning with the 2013-14 administration of the NYSAA in 
ELA and mathematics. 

To align the ELA and mathematics NYSAA with the CCLS, the Department began work in 
November 2011 to develop an alternate assessment to measure the CCLS for ELA and mathematics 
for students with severe cognitive disabilities. The Department brought together special education and 
general education teacher committees, including general education content specialists and special 
education teachers, to review the CCLS for the content identified in the new Test Blueprint, develop 
Essence statements to narrow the depth and breadth of the CCLS, and draft Extensions. The new Test 
Blueprint for the NYSAA was based on the general education Test Blueprints to determine the 
academic core priorities. The Test Blueprints for the NYSAA that measure the Extensions to the CCLS 
in ELA and mathematics were approved in spring 2012. The draft Essences and Extensions were 
reviewed extensively during the summer of 2012, and draft documents were posted for public comment 
in September 2012. In October 2012, the committees were reconvened to review the revisions to the 
Essences and Extensions, and to draft Assessment Tasks to measure student performance of the 
CCLS. Following the meeting, the draft Assessment Tasks were reviewed and vetted by content and 
Special Education Teachers, and then were posted for public comment from December 2012 to 
January 2013. Public comments from the first review and the second review were incorporated, as 
appropriate, into the draft Extensions and draft Assessment Tasks. 

The administration procedures for the NYSAA were revised to a new test design that 
emphasized a continuum of student performance and intentionally focused on increasing the validity of 
test scores. The procedures were streamlined, where possible, to reduce teacher clerical errors. The 
2013-14 administration procedures applied to all content areas. This was done so that teachers would 
not have to use two completely different assessment procedures. The new NYSAA was first 
implemented in 2013-14. The administration culminated with regional Scoring Institutes. Standard 
setting was conducted in June 2014, resulting in cut scores for each grade level and content area, as 
well as in APLDs. The cut scores were approved by the Commissioner of Education, following standard 
setting. 

The information in this section and throughout the Technical Report provides a framework to 
determine the consequential validity of the NYSAA. In order to demonstrate consequential validity, the 


assessment should: 


= provide multiple measurement occasions, 
» show student results are improving, and 


= demonstrate that revisions to the NYSAA are considered based on Stakeholder 
feedback. 
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The revised NYSAA demonstrates that students are provided multiple measurement occasions 
as embedded in the baseline and final data collection points. Also, Stakeholder input has been critical 


throughout the development and revision processes. 
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APPENDIX A—NYSAA TEST BLUEPRINTS FOR 
EACH CONTENT AREA 
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New York State Alternate Assessment (NYSAA) 


Test Blueprint for English Language Arts (ELA) 


ENGLISH LANGUAGE 
ARTS (ELA) 


Strand 


Sub-Strand 


Grade 
3 


Grade 
4 


Grade 
5 


Grade 
6 


Grade 


Grade 
8 


High 
School 


Reading Standards for 


Literature 


Reading Standards for 
Informational Text 


Writing Standards 


Speaking 


and 


Language 


Standards 


Listening 


Key Ideas and Details 


X 


Craft and structure 


Integration of 
Knowledge and Ideas 


Responding to Literature 


Key Ideas and Details 


Craft and Structure 


Integration of 
Knowledge and Ideas 


Key Ideas & Integration 
of Knowledge and Ideas 


Text Types and 
Purposes 


Production and 
Distribution of Writing 


Research to Build and 
Present Knowledge 


Comprehension and 
Collaboration 


Presentation of 
Knowledge and Ideas 


Conventions of Standard 
English 


Knowledge of Language 


Vocabulary Acquisition 
and Use 
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New York State Alternate Assessment (NYSAA) 
Test Blueprint for Mathematics 


Grade | Grade | Grade | Grade | Grade | Grade | High 


Domain 3 4 5 6 7 8 School 
Operations and Algebraic 

Thinking (OA) Xx x xX 

Number and Operations in Base 

Ten (NBT) xX x Xx 

Number and Operations — 

Fractions (NF) Xx Xx Xx 

Measurement and Data (MD) X Xx Xx 

Geometry (G) Xx Xx Xx 


Ratios and Proportional 
Relationships (RP) 


The Number System (NS) 


Expressions and Equations 
(EE) 


Functions (F) 


Statistics and Probability (SP) 


Quantities (NQ) 


xX 
Creating Equations (A-CED) x 
Interpreting Functions (F-IF) X 
Expressing Geometric 
Properties with Equations (G- X 
GPE) 
Interpreting Categorical and 
Quantitative Data (S-ID) Xx 
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NYSAA Test Blueprint - Science 
Effective with 2013-14 Administration 


Two Standards are assessed for each Grade as Marked by an X 


Grade Grade High 
Standards Chapter 5. Key Idea A 8 ae 


2- Beyond the use of reasoning and 
consensus, scientific inquiry 
involves the testing of 
explanations involving the use of 

; ; conventional techniques and 

1 — Analysis, Inquiry, procedures and usually requiring 


and Design (Scientific considerable ingenuity. 
Inquiry) 


testing proposed explanations, 
when analyzed using conventional 
and invented methods, provide 
new insights into phenomena.. 


Living things are both similar to 
and different from each other and 


4- Living Environment from nonliving things. 


3- Individual organisms and species 
change over time. 


Many of the phenomena that we 
observe on Earth involve 
interactions among components of 
4- Physical Setting/ air, water, and land. 

Earth Science 3- Matter is made up of particles 
whose properties determine the 
observable characteristics of 
matter and its reactivity. 


*Note: See the Core Curricula for Science at http://www.p12.nysed.gov/ciai/cores.htmlI#MST. 
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NYSAA Test Blueprint - Social Studies (HS only) 
Effective with 2013-14 Administration 


Two Standards are assessed for each Grade as Marked by an X 


High 


nit 
Standards U nits Scheel 


1- US History 2 - Constitutional Foundations X 


2- World History: 
Global History 8 - Global Connections and Interactions 
and Geography 


See the Core Curricula for Social Studies 


at: http://www.p12.nysed.gov/ciai/cores.htmlIHSOCIALSTUDIES 


2013-14 NYSAA Technical Report: Appendix A—NYSAA Test Blueprints for Each Content Area -6- 


APPENDIX B—NYSAA Scoring Procedures 
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Procedures for Scoring NYSAA Datafolios | 2013-14 


e Follow the steps outlined below to review each NYSAA datafolio. 
e Review the documentation to determine the answer to the question/statement for each step. 
e If a discrepancy is not addressed in this document, consult your Table Leader. 


e Prior to the Scorer recording the error a Table Leader MUST review and confirm all issues that may 
result in a “No” for any of the three Connections questions, a “No Score” for a date(s) and/or an 
adjustment(s) to the Data Summary Sheet (DSS). 


1. Student Demographics, Scorer ID, Scoring Institute Code 
a) Is the student demographic information consistent? 


Note: Student demographic information must be consistent between the demographic label (from the RIC), 
Student Page (in datafolio), and Scannable Score Document. If discrepant or if scannable is missing, consult 
the Table Leader. Record Scorer comment A at the bottom of the Scorer Worksheet. 


b) Apply label in the upper left corner on each page of the Scorer Worksheet. If a label is not 
available, transcribe the information from the Student Page to the Scorer Worksheet. 


c) Fill in your Scorer ID and the Scoring Institute Code. 


Enter your 3-digit Scorer Identification Number and 6-digit Scoring Institute Code in the upper right corner of the 
Scorer Worksheet. 


d) Does the DOB fall within the range indicated on the Student Page for the grade assessed? 
i. If Measured Progress ProFile™ used - accept the grade level as correct. 


ii. | Note: If a DOB is found to be outside the range specified for any grade level, consult your 
Table Leader. 


lf YES > Mark the grade assessed in the upper right corner of the Scorer Worksheet. 


lf NO > Wrong grade level was assessed. Record: 

Consult e Extension/AGLI code 00099 

the Table e “N’ for No for all Connections questions for each 
Leader Extension/AGLI within the content areas that should 


have been assessed 
e “NS” for No Score for baseline and final dates 
e Procedural Error comment 1 


e) Are there any Testing Accommodations listed on page two of the Student Page? 
Note: If page two of the Student Page is missing or incomplete, continue to review and score the assessment. 


lf YES > | Transcribe any Testing Accommodations to the Scannable Score Document. 
lf NO > Continue to review and score the assessment. 


f) Was a Collegial Review month indicated on the Student Page? 


If YES > Record: 
e “Y” for Yes for “Was a collegial review of this datafolio conducted?” on the Scannable Score 
Document. 
lf NO > Record: 
e “N’ for No for “Was a collegial review of this datafolio conducted?” on the Scannable Score 
Document. 
e Scorer comment B, at the bottom of the Scorer Worksheet 
Continue to review and score the assessment. 


g) Set aside the Scannable Score Document until all content areas have been reviewed and 
scored. 


2. Review Sequence of Documentation for each Content Area 
ELA, mathematics, science, and social studies. Review entire content area to determine if anything is out of 
order. Do not reorganize the datafolio. 


a) Are required DSSs present, one for each Standard assessed? (refer to DSS Titles [e.g., Looking at 
ELA and Math do you have 1, 2, 3, 4, and 5 in titles; in Science and Social Studies do you have 1 and 
2)) 


lf NO > DSS missing. Record: 

Consult e Extension/AGLI code 00099 

the Table e “N’ for No for each Connection question for the missing 
Leader Extension/AGLI 


e “NS’ for No Score for baseline and final dates 
e Procedural Error comment 2 
Proceed to the next Extension/AGLI that has a DSS or next 
content area if all DSSs are missing. 
Two or more DSSs for the same | e Review and score the first DSS and assessment 


Standard and there are fewer documentation for the Standard assessed 
than the required number of For the missing Standard, record: 
Standards assessed. e Extension/AGLI code 00099 


e “N’ for No for each Connection question 

e “NS” for No Score for baseline and final dates 

e Procedural Error comment 5 

Proceed to next Extension/AGLI or the next content area. 


b) Are the DSS forms in order? 
Confirm that the DSSs are in the correct order using the titles (upper right corner) on each form. 


lf NO > Documents are out of order. e Consider documentation that is out of order and score the 
Consult assessment in the correct order. Do not reorganize 
the Table datafolio. 

Leader Record: 


e Scorer comment C 
Proceed to Step 3. 


3. Is demographic information on DSS complete and accurate when compared to the 
Student Page? 


lf NO > Demographic information is e Transcribe information from the Student Page to the DSS in 
Consult discrepant or incomplete. red ink. 

the Table Record: 

Leader e Scorer comment D 


Proceed to Step 4. 


4. Extension/AGLI from Grade Level 
Is the Extension/AGLI indicated on the DSS from the student’s assessed grade? 
lf YES > |e Extension/AGLI is from grade Record: 


or e Extension/AGLI code (5-digits) 
e Measured Progress ProFile™ e “Y” for Yes for “Extension/AGLI from Grade” 

used to complete datafolio Proceed to Step 5. 

documentation 
If NOT Extension/AGLI is missing on the | If Extension/AGLI code or text is found on the VE, and 
SURE > DSS but can be located on VE code/text matches the Frameworks, transcribe the information 
Consult to the DSS in red ink and continue to review and score the 
the Table assessment. 
Leader Record: 


e Extension/AGLI code (5-digits) 
e “Y” for Yes for “Extension/AGLI from Grade” 
e Comment D 
Proceed to Step 5 
If NO7> DSS for a standard includes an Record: 


Consult Extension/AGLI from an e Extension/AGLI code 00099 

the Table | inappropriate grade level e “N” for No for each Connection questions 

Leader e “NS” for No Score for baseline and final dates of the 
Extension/AGLI 


e Procedural Error comment 4 
Proceed to next Extension/AGLI or content area. 


5. Task Connects to Extension/AGLI 


Does the Assessment Task documented on the DSS clearly connect to the Extension/AGLI? (reference 
Extension and AT codes) 


If the Assessment Task is hand written, Scorer must verify against the AT text in the Frameworks to ensure text 
is exact and AT is from the same Level of Complexity. 


lf YES > |e Assessment Task clearly Record: 
connects to selected e “Y” for Yes for “Task Connects to Extension/AGLI" 
Extension/AGLI or Proceed to Step 6 


e Measured Progress ProFile™ 
used to complete DSS 


If NOT Assessment Task is missing on If Assessment Task code or text is found on the VE, and 
SURE > the DSS but can be located on code/text matches the Frameworks, transcribe the information 
Consult VE to the DSS in red ink and continue to review and score the 
the Table assessment. 

Leader Record: 


e “Y” for Yes for “Task Connects to Extension/AGLI" 
e Comment D2 
Proceed to Step 6 


If NO7> Assessment task does not Record: 

Consult connect to Extension/AGLI e “N” for No for “Task Connects to Extension/AGLI” and 

the Table remaining Connection question 

Leader e “NS” for No Score for baseline and final dates of the 
Extension/AGLI 


e Procedural Error comment 6a 
Proceed to next Extension/AGLI or content area. 
Assessment task is missing and | Record: 


cannot be located on the VE e “N’ for No for “Task Connects to Extension/AGLI’ and 

(either evidence itself or VE remaining Connection question 

label). e “NS” for No Score for baseline and final dates of the 
Extension/AGLI 


e Procedural Error comment 2 
Proceed to next Extension/AGLI or content area. 


6. VE Connects to Task (confirming VE connected only, not whether VE valid) 
a) Is evidence for both the baseline and final data point found behind the DSS? (two pieces of VE) 


Note: A single DCS may be considered as two pieces of VE. A calendar/chart can only be submitted as one 


piece of VE. 


If MORE 
THAN 
TWO 
PIECES 
OF VE > 


If NO > 
Consult 
the Table 
Leader 


If more than baseline and 
final VE are included 


Note: Do not confuse this with 
a Student work product that is 
multiple pages, or with 
supporting evidence. 


Only one piece of evidence 

is found 

Note: Review the entire 
datafolio to determine if second 
piece of VE is misplaced. |f VE 
is misplaced, leave it where 
found, review and score. 


e Only evidence for the two dates (baseline and final) 
documented on the DSS can be considered. 

e Also, if one or both pieces of evidence for this 
Extension/AGLI are found to be invalid (Step 8), other 
evidence cannot be considered in its place. 

e If date(s) on VE are discrepant with DSS, use VE for earliest 
date as baseline and latest date for final. 

Record: 

e Scorer comment G 

Proceed to Step 7b using only VE for baseline and final. 

Record: 

e “N’ for No for “VE connects to task” 

e “NS” for No Score for baseline and final dates 

e Procedural Error comment 7 

Proceed to next Extension/AGLI or content area. 


b.) Does the evidence for each data point connect to the Assessment Task documented on DSS? 


e Data Collection Sheets (DCS) must include step, trial or time segment information 
e Digital video and audio evidence must be accessible and can be reviewed by Scorer 
e Toconnect, each piece of evidence, on its own, must: 
v Meet the intent of task by demonstrating the student's skill on the assessed task 
v¥ Not include information (e.g., directions, items) that conflicts with the vocabulary of task (e.g., 
main character = main character, main character # important character; reference content 
glossary to confirm) 
¥ Demonstrate any plural in the task (if parenthesis on “s,” plural requirement is optional) 
v Demonstrate any AND in the task by demonstrating all elements of the assessed task 


If YES > 


If NO > 
Consult 
the Table 
Leader 


Evidence for both the baseline 
and final data point 
demonstrate the above 
criteria on their own 

Evidence for baseline and/or 
final data point does not 
connect to Assessment Task 


Assessment Task includes a 
plural without parentheses 
around “s”. Upon review, one 
or both pieces of VE do not 
satisfy the plural. 


Assessment Task includes an 
AND statement. Upon review, 
one or both pieces of VE do 
not satisfy the AND 
statement. 


Proceed to Step 6c. 


Record: 

e “N’ for No for “VE Connects to Task” 

e “NS” for No Score for baseline and final dates of the 
Extension/AGLI 

e Procedural Error comment 8a, 8b, or 8c 

Proceed to next Extension/AGLI or content area. 

Record: 

e “N’ for No for “VE Connects to Task” 

e “NS” for No Score for baseline and final dates of the 
Extension/AGLI 

e Procedural Error comment 8d 

Proceed to next Extension/AGLI or content area. 

Record: 

e “N’ for No for “VE Connects to Task” 

e “NS” for No Score for baseline and final dates of the 
Extension/AGLI 

e Procedural Error comment 8d 

Proceed to next Extension/AGLI or content area. 


DCS included as VE is 
missing the step, trial or 
time-segment information 


DCS includes a single step or 
time-segment that does not 
clearly document the student 
performance for the assessed 
task (e.g., plural or AND) 


Digital video and/or audio 
malfunctioned or the clip is 
unable to be located on the 
DVD and/or CD 


Record: 

e “N’ for No for “VE Connects to Task” 

e “NS” for No Score for baseline and final dates of the 
Extension/AGLI 

e Procedural Error comment 16d 

Proceed to next Extension/AGLI or content area. 

Record: 

e “N’ for No for “VE Connects to Task” 

e “NS” for No Score for baseline and final dates of the 
Extension/AGLI 

e Procedural Error comment 3 

Proceed to next Extension/AGLI or content area. 

Record: 

e “N’ for No for “VE Connects to Task” 

e “NS” for No Score for baseline and final dates of the 
Extension/AGLI 

e Procedural Error comment 15d 

Proceed to next Extension/AGLI or content area. 


c.) Is the Level of Accuracy, for the baseline data point, 74% or below? (threshold) 


If YES > 


If NOT 
SURE > 
Consult the 
Table 
Leader 


If NO > 
Consult 
the Table 
Leader 


On review of the VE for 
baseline, Level of Accuracy 
score is confirmed to be 74% 
or below. 


Information on the VE 
contradicts or does not 
support what is documented 
for the Level of Accuracy 


Level of Accuracy score for 
baseline is 75% or higher (as 
documented or recalculated) 


Record: 


e “Y” for Yes for “VE Connects to Task” 
Proceed to Step 7. 


e = If error in calculation is clear 
For example: 


Math example: “4 + 2 = 6” is marked incorrect by the teacher 


but is clearly correct. 
Adjust the baseline score on the DSS in red ink and if 


recalculation forces baseline score to 75% or higher, proceed 


to No below. 


e If scorer disagrees, or correct answer can be debated but 
cannot be clearly resolved one way or the other, accept 
the percentage documented. Scorer comment K3 or N. 


Proceed to Step 7. 

Record: 

e “N’ for No for “VE Connects to Task” 

e “NS” for No Score for baseline and final dates of the 
Extension/AGLI 

e Procedural Error comment 8e 

Proceed to next Extension/AGLI or content area. 


7. Dates of Student Performance on the DSS 


Are dates recorded on the DSS for the baseline and final data points within the 2013-14 NYSAA 
administration period (September 30, 2013—February 7, 2014)? 


If NO > One or both dates of student e Transcribe date(s) from VE to the DSS in red ink. 


Consult performance within the Record: 
the Table administration period is e Scorer comment J 
Leader missing from DSS, but can be 
determined from VE. 
One or more dates of student Record: 
performance within the e “NS” for No Score for date(s) in question 
administration period cannot e Procedural Error comment 11a or 11b 
be determined from VE or one | Review the remaining date(s), proceed to next Extension/AGLI, 
or more dates on DSS are or content area. 
outside the administration 
period. 


8. Valid Evidence: VE and Supporting Evidence (SE) 
a) Are the THREE required elements clearly documented on each piece of VE and match 
information on the DSS? (Verifying Evidence and Supporting Evidence) 


Required elements may be handwritten or printed on the actual VE, on a VE label that is affixed to the VE, ora 
combination of both. A student may record his or her name and/or the date on work products. It is acceptable 
for only the student’s first name to be documented on the VE. 


lf YES > Student name, date of student Proceed to Steps 8b, c, e, and/or f depending on type of 


performance and Level of evidence. 
Accuracy are clearly documented. 
If NO > One or more required elements on e Adjust the required element(s) on the DSS in red ink. 
Consult the VE and/or VE label is Record: 
the Table discrepant with the DSS. e Scorer comment K1, K2, or K3 
Leader Note: Use chart below Proceed to Steps 8b, c, e, and/or f. 


Note: Do not make any marks on VE or VE labels 


The following ... 


Required elements documented by the teacher on the VE.......... The DSS and the VE label 
Required elements on the VE label..........................:::0eeeeee es The DSS 

Teacher recorded information. ...................0.ccececeeseeeeeneaeeeees Student recorded information on VE 
lf NO > One or more required elements is Record: 


Consult missing from VE (VE itself or VE e “NS” for No Score for that date 

the Table label) or VE label is not affixed to e Procedural Error comment 12a, 12b, or 12c¢ 

Leader VE. Review remaining date, or proceed to next Extension/AGLI, 
or content area. 


b) Is the student work product original? (Student Work Product) 


e No photocopies of student responses, correction fluid/tape or white/black out. 
e Students may use assistive technology, computers, and/or interactive white board systems (e.g., 
SMART board) to complete the student work product. 


If YES > Continue to review the other piece of VE submitted or proceed to Step 9. 
If NO > Work product is not original (i.e., Record: 


Consult photocopies of student responses, e “NS” for No Score for that date 
the Table correction fluid/tape, black out). e Record Procedural Error comment 13 
Leader Review remaining date, or proceed to next Extension/AGLI, 


or content area. 


c) Does the DCS have SE and at least three dates of data with staff initials for each date 


recorded? 
The DCS must have 


a minimum of three dates of documented student performance, 

one piece of SE for each date transcribed to the DSS as VE, 

staff initials recorded for each date on the DCS, and 

Supporting evidence may be an Observer Verification Form OR another type of VE 


o NOTE: It is acceptable for DCS to be considered as VE for only the baseline, only the final or both 
the baseline and final data points. In all cases, the DCS must include at least 3 dates of data and 


meet all other requirements. 


If YES > 

lf NO > Fewer than three dates are 
Consult documented on the DCS. 
the Table 

Leader 


SE is missing for either date 
transcribed to DSS as VE. 


Staff initials are missing from 
DCS for any date 


d) Is the SE valid? (Supporting Evidence) 


Continue with Step 8d below; review each submitted piece of SE individually. 


Record: 

e “NS” for No Score for any date on the DCS 
transcribed to the DSS 

e Procedural Error comment 16a 

Review remaining date, or proceed to next Extension/AGLI, 

or content area. 

Record: 

e “NS” for No Score for the date(s) transcribed from the 
DCS to the DSS 

e Procedural Error comment 16c 

Review remaining date, or proceed to next Extension/AGLI, 

or content area. 

Record: 

e “NS” for No Score for any date the DCS transcribed to 
the DSS 

e Procedural Error comment 16b 

Review remaining date, or proceed to next Extension/AGLI, 

or content area. 


1. Student Work Product - Review Steps 8a and b to determine if student work product is valid SE. 
2. Photographs - Review Steps 8a and e to determine if photographs are valid SE. 
3. Digital video and/or audio clip - Review Steps 8a and f to determine if digital video and/or audio 


clip is valid SE. 
lf YES > 
lf NO > Student work product, photographs, 
Consult or digital video or audio clip is 
the Table | invalid per Step criteria. 
Leader 


4. Observer Verification Form (OVF) 


Continue to review the other piece of SE submitted or proceed to Step 9. 


Record: 

e “NS” for No Score for that date 

e Appropriate Procedural Error comment indicated in 
Steps 8a, b, e, orf 

Review the other piece of SE submitted, for remaining date 

or proceed to next Extension/AGLI or content area. 


o Review Step 8a and OVF criteria below to determine if OVF is valid SE. 
o NOTE: Only a DCS requires SE. Ignore an OVF submitted in support of original student work, 
photographic, digital video or audio evidence. 


Criteria for an OVF 
An OVF is invalid if: 
* Student name, date of student performance, and/or Level of Accuracy are missing; 

* supplementary school personnel signed as the observer (e.g., teacher aide or teacher assistant); 

¢ the person collecting the data (initials on DCS) also signed the OVF as the observer for that date (confirmed 


by comparing initials and staff key information); 
more than one date of student performance is documented on a single OVF; 
the observer's signature and/or title is not included, cannot be confirmed; 
the observer's signature date is missing, or is not the same date task was observed; or 
date of performance on OVF is discrepant with date of performance on DCS 


If NO > 
Consult the 
Table 
Leader 


OVF is invalid per one or more 
criteria listed in the bullets above. 


Observer’s title is missing from 
OVF, but can be confirmed from 
another OVF in the datafolio. 


Observer’s title is missing from 
OVF and cannot be confirmed 
from another OVF in the datafolio. 


Record: 

e “NS” for No Score for that date 

e Procedural Error comment 12a—c or 17a-e 

Review the other piece of SE submitted for remaining date 
or proceed to next Extension/AGLI or content area. 

Score the assessment. 

Record: 

e Record Scorer comment M 

Continue to review the other piece of SE submitted or 
proceed to Step 9. 

Record: 

e “NS” for No Score for that date 

e Procedural Error comment 17b 

Review the other piece of SE submitted for remaining date 
or proceed to next Extension/AGLI or content area. 


e) Are there three photographs with a caption from a single date? (Photographs) 
Photographic evidence must be 
e aminimum sequence of three photographs of the student performing the task, 
e aminimum of one caption describing the sequence, and 
e the sequence must occur on a single date. 


If YES > 
If NO > 
Consult 
the Table 
Leader 


Continue to review the other piece of VE submitted or proceed to Step 9. 


Fewer than three photographs 
are submitted of the student 
performing the task. 


No caption is found. 


No date or multiple dates are 
found on the evidence. 


Record: 

e “NS” for No Score for that date 

e Procedural Error comment 14d 

Review remaining date, or proceed to next Extension/AGLI, 
or content area. 

Record: 

e “NS” for No Score for that date 

e Procedural Error comment 14c 

Review remaining date, or proceed to next Extension/AGLI, 
or content area. 

Record: 

e “NS” for No Score for the date 

e Procedural Error comment 14a or 14b 

Review remaining date, or proceed to next Extension/AGLI, 
or content area. 


f) Is the digital video and/or audio clip brief and does it contain the three required elements 
recorded? (Digital Video/Audio Clip) 


Clip must be 


e 90 seconds or fewer (excluding markers) and 
e contain at least one recorded marker with the Student’s name, date of student performance and 
Level of Accuracy. 


If YES > 


If NO > 
Consult 
the Table 
Leader 


Continue to review the other piece of VE submitted or proceed to Step 9. 


Clip duration is longer than 90 
seconds and it is unreasonable to 
review entire clip. 


Student’s name, date of student 
performance and Level of Accuracy, 
are not recorded on the clip in any 
manner. 

Note: VE label on DVD/CD case or 
box is not acceptable. 


Record: 

e “NS” for No Score for that date 

e Procedural Error comment 15c 

Review remaining date, or proceed to next Extension/AGLI, 
or content area. 

Record: 

e “NS” for No Score for that date 

e Procedural Error comment 15a 

Review remaining date, or proceed to next Extension/AGLI, 
or content area. 


9. Were any supports provided that guided the student to the correct answer? 
Scorers must review each piece of VE to consider whether any documentation guided the student to the correct 
answer.. Refer to the chart below for details. 


Actions That Result in an Administrative Error 


Templates or other formats are provided that give or lead the student to the answer. For 


example: 


e the verifying evidence is a sequencing worksheet that contains three boxes that 
state “First,” “Next,” “Last”; the student response choices are pictures that 
contain the words “First,” “Next,” “Last.” 

e the verifying evidence is a number line on which the student must provide 
missing numbers, but the correct number is provided as a shaded or dotted 
number in the spot and the student has to put a sticker of the number on the 


spot. 


If YES > 


If VE for baseline and/or final data 


Consult the point includes documentation 


Table 
Leader 


If NO > 


which led the student to the 
answer 


Record: 

e “NS” for No Score for that date 

e Procedural Error comment 18a or 18b 

Review remaining date, or proceed to next Extension/AGLI, 
or content area. 


Continue to review and score the assessment 


10. Is the Level of Accuracy documented on DSS, for the final data point, calculated 
correctly based on VE? (Baseline Level of Accuracy was checked at Step 6c) 


If YES > 


If NOT 
SURE > 
Consult the 
Table 
Leader 


If NO > 
Consult 
the Table 
Leader 


Record: 


e Percentage for the Level of Accuracy for both baseline and final dates 
e Yes or No for “was student prompted?” as documented on DSS for both baseline and final 


date. Note: do not verify prompts. 


Information on the VE contradicts 
or does not support what is 
documented for the Level of 
Accuracy, and the Scorer cannot 
clearly see how to correct 
calculation. 


Level of Accuracy is missing from 
the DSS, but is present for a date 
that has valid VE. 


Level of Accuracy on the VE is 
discrepant with what is 
documented on the DSS. 


Level of Accuracy was incorrectly 
calculated and the Scorer can 
clearly see how the percentage 
calculated can be adjusted. 

Note: if Scorer cannot clearly see 
how to correct calculation, follow “If 
NOT SURE >’ directions. 


e If error in calculation is clear 
For example: 


Math example: “4 + 2 = 6” is marked incorrect by the 
teacher but is clearly correct. 


Adjust the baseline score on the DSS in red ink and if 
recalculation forces baseline score to 75% or higher, 
proceed to No below. 


e If scorer disagrees, or correct answer can be debated 
but cannot be clearly resolved one way or the other, 
accept the percentage documented.. 


Accept the percentages the teacher documented. 
Record: 


e Percentage documented by the teacher 
e Percentage for baseline 


e Yes/No for the question “was the student 
prompted?” as documented on DSS for both 
baseline and final 

e Scorer comment K3 or N 

Proceed to Step 11 

e Transcribe percentage calculation from the VE to the 
DSS in red ink. 

Record: 

e Percentage from the VE 


e Yes/No for the question “was the student 
prompted?” as documented on DSS 


e Scorer comment D4 


e Adjust the percentage calculation on the DSS in red ink 


to match the VE. 
Record: 
e Adjusted percentage 
e Scorer comment K3 
Note: Never make changes to VE or VE labels. 


e Adjust the percentage calculation and/or rating on DSS 
in red ink. 

Record: 

e Adjusted percentage 

e Scorer comment O 


11. Score the next Extension/AGLI 
Follow Steps 3-10 for the next Extension/AGLI from the same content area. 


12. Score Mathematics, Science, and Social Studies 


Follow Steps 2—11 and score 
(Grades 3-8 & HS), science ( 


the remaining content areas in order for the grade assessed: mathematics 
Grades 4, 8, & HS), and social studies (HS only). 


13. Confirm your Scorer Worksheet is complete for each Extension/AGLI within each 
content area including Procedural Error Comments, if applicable and Scorer Comments 
e Extension/AGLI Code, Three Connections Questions - Double check that a five digit 
Extension/AGLI code has been recorded; that the three Connections questions are bubbled in as 
“Y” or “N”; and percentages for Levels of Accuracy are recorded for both baseline and final data 


points. 


e Procedural Error Comments (1-20) - Double check that a Procedural Error Comment has been 
recorded on the Scorer Worksheet for each No or No Score. 


e Scorer Comments (A-O) / Positive Feedback Comments (P-W) - Select comments from the back 
of the Scorer Worksheet that will clarify if something was adjusted in the datafolio and/or if something 


was questioned duri 
teachers. 


ng scoring. Scorers are encouraged to also provide positive feedback to 


e No blank spaces unless the content was not assessed. 
e Confirm Extension/AGLIs have been recorded correctly — each code is documented in the correct 


space. 


14. Complete the Scannable Score Document 


Transcribe the following data: 


From the: e Extension/AGLI code - 5 digits 
e Scorer e Three Connections questions - “Y” for Yes or “N” for No 
Worksheet fo) Extension/AGLI from grade level 
Oo Task connects to Extension/AGLI 
fe) VE connects to task 
e Percentages - Level of Accuracy for baseline and final 
e Yes/No for “was student prompted?” 
From the: e Absent 
e “Not Tested” e Administrative Error 
form, if e Not Enrolled 
applicable e Took Another Assessment 


e Medically Excused 


Confirm you have c 


ompleted: 


From the: 
e Student Page 


e Was a Collegial Review of this datafolio conducted? “Y” for Yes or “N” for No 


e Transcribe the Testing Accommodations documented on page 2 of the 
Student Page to the Scannable Score Document in the space provided. 


e Complete the Scannable Score Document for each applicable content area and for any other 
information as directed by the SSC. 


CAUTION - 
Performance 
Document wi 


Errors in transcribing Connection to Grade Level Content and 
ratings from the Scorer Worksheet to the Scannable Score 
ll directly impact the student’s receiving a reportable score. DOUBLE 


CHECK ALL TRANSCRIPTIONS TO THE SCANNABLE SCORE DOCUMENT! 


O] at-ymeotere) alate MOxeyaler-1 sat) 


vo) am @TUC-¥-3 tro) at-wy (oj mr-(eolela-t-t-\-1e mel a-Xerd NVM aM tal: Melero) alate Ml ad weler-Lelela-t-) 


This table outlines other issues that may come up when scoring a datafolio. These may result in a No Score 
and/or adjustment to the datafolio. If any of these issues are found, consult the Table Leader for direction. 


The following may or may not result in a No or No Score. 


e Incorrect or teacher-created NYSAA forms were used (e.g., Data Summary Sheet (DSS) for the wrong grade, 
Student Page from 2012-13, or Data Collection Sheet not from NYSAA Administration Manual). 


e Student Page is missing 

e Photocopies (either in part or whole) or correction fluid/tape or black out is found on assessment documents. 
e “By” statement from Assessment Task not demonstrated on VE 

e Level of Complexity, task assessed, is different between the baseline and final data points 


e Evidence is found that a mistake in data collection was erased on the DSS, VE, or supporting evidence and 
was not crossed out and initialed by the teacher. 


e VE or supporting evidence clearly appears to be homework. 
e VE for ELA is submitted in a language other than English. 
e VE includes information that leads or guides the student toward the correct answer (e.g., template). 
The following may occur in a datafolio and are acceptable, providing they meet requirements. 


e Presentation or number of items is different between the baseline and final data point 
e Chart or calendar is submitted for a date other than the last date recorded on the chart or calendar. 


e The DCS includes steps not relevant to the assessed task, or a single-step task is documented on a multi-step 
DCS. 


e Extra VE or supporting evidence was submitted beyond the requirements for a specific Extension/AGLI. 
e Extra DSSs are found 


e Dates or information printed in the header and/or footer of documents completed with Measured Progress 
ProFile™ contradict information recorded on the evidence or VE label. 
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APPENDIX C—2013-14 SCORING 
DECISION RULES 


2013-14 NYSAA Technical Report: Appendix C—2013-14 Scoring Decision Rules 


Decision Rules for Scoring NYSAA Datafolios 


(For Table Leaders) 


2013-14 


® 
S 
a 

1 


Scoring 
Concern/Question 


Incorrect or teacher-created 
NYSAA forms were used (e.g., 
Data Summary Sheet (DSS) for 
the wrong grade, Student Page 
from 2012-13, or Data Collection 
Sheet not from NYSAA 
Administration Manual). 


Student Page is missing 


Photocopies (either in part or 
whole) or correction fluid/tape or 
black out is found on assessment 
documents. 


Assessment Task 


4 


Assessment Task includes a “by” 
statement which is not 
demonstrated in the VE 


Level of Complexity (a different 
Assessment Task) changes 
between baseline and final data 
points 


Verifying Evidence (VE) 


6 


Presentation or number of items 
differs between the baseline and 
final VE 


May 


Decision Rule/Rationale in eee 


Step(s) 


Incorrect Forms 1-8 
e lfan incorrect Student Page or DSS is used but all assessment 
requirements can be confirmed, score the assessment following the 


Scoring Procedures. 


e If anincorrect DSS is used and assessment requirements cannot be 
confirmed, record Extension/Alternate Grade Level Indicator (AGLI) 
code(s) 00099 and “N” for No for each Connections question and “NS” 
for No Score for baseline and final of the Extension(s)/AGLI(s). Record 
Procedural Error comment 19. Continue to next Extension/AGLI or content 
area. 


Teacher-created Forms 


e Teacher created his/her own 2013-14 forms, such as a Data Collection 
Sheet or VE label. If all requirements are clearly documented, score the 
assessment following the Scoring Procedures. 


If the student demographic information (student name, date of birth) on the 1 
DSS can be used to confirm the correct student was assessed continue to 
review and score the datafolio. Direct the Scorer to record comment A. 


e Correction fluid/tape or black out found on page numbers, Student Page, or | 1-8 
table of contents does not directly impact scores. Score the assessment 


following the Scoring Procedures. 


e Photocopies of the DSS, VE, or supporting evidence (either in part or in 
whole) or correction fluid/tape or black out found on information will directly 
impact student scores. 


o If DSS, record “NS” for No Score for baseline and final dates. 
o If VE, record “NS” for No Score for that date. 
o Record Procedural Error Comment 13 


Note: Digital photo prints in color or in black and white, computer/tablet device 
printouts, and interactive white board (e.g., SMART board) printouts are 
acceptable, since they are not photocopies. 


If any part of the Assessment Task, including a “by” statement, is not 6 
demonstrated in the Verifying Evidence, and there is no teacher notation to 

clarify, record “N” for No to “VE connects to Task” and “NS” for No Score 

for baseline and final of the Extension/AGLI. Record Procedural Error 

Comment 8. Continue to the next Extension or AGLI. 


The exception to this rule is when the teacher has indicated a method of 
response and the student demonstrates a different method of response. This 
type of by statement is not related to the Assessment Task and different 
method of response would be acceptable. 


If the baseline administration is from one Level of Complexity and the final 
administration is from another Level of Complexity within the same Standard 
(student was not given the baseline again, when Level of Complexity 
changed), record “N” for No to “VE connects to Task” and “NS” for No 
Score for baseline and final dates. Record Procedural Error Comment 10. 


Best practice for administration was to provide similar format and presentation | 6 
across the baseline and final administration. For scoring 2013-14, as long as 

the connection of “VE to task” is clearly demonstrated, any change in format, 
presentation, number of items, etc. should be ignored. Continue to review and 
score the assessment. Direct the Scorer to record comment F. 
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10 


11 


12 


13 


Evidence is found that a mistake 
in data collection was erased on 
the DSS, VE, or supporting 
evidence and was not crossed 
out, corrected, and initialed by the 
teacher. 


VE or supporting evidence 
clearly appears to be homework. 


VE for ELA is submitted in a 
language other than English. 


Chart or calendar is submitted for 
a date other than the last date 
recorded on the chart or 
calendar. 


VE appears to include a format 
that guides the student directly 
to the correct answer (e.g., 
template). 


Photographic, digital video, or 
audio evidence appears to 
include prerequisite or post- 
activity steps. 


The DCS includes steps not 
relevant to the assessed task, a 
single-step task, or single time 
segment is documented ona 
DCS. 


A student may self-correct on a student work product, which does not 
require a notation by the teacher. 


e If ateacher-made error is crossed out and corrected but not initialed, score 
the assessment following the Scoring Procedures. 


e lf ateacher-made erasure is confirmed, record “NS” for No Score for that 
date. Record Procedural Error comment 13. Continue to review and score 
other date for the Extension/AGLI following the Scoring Procedures. 

Note: Documentation made by the teacher does not have to be in permanent 

ink. 

e If the Student Page indicates special education programs and services at 
home, in a hospital, or other facility, accept what is documented by the 
teacher and score the assessment following the Scoring Procedures. 


e If the Student Page does not indicate special education programs and 
services at home, in a hospital, or other facility, record “NS” for No Score 
for that date. Record Procedural Error comment 20. Continue to score next 
date. 


Record “NS” for No Score for that date. Record Procedural Error comment 
21. Continue to score next date. 


A chart or calendar can be submitted for only a single date. If the date on the 
calendar or chart is within the administration period for the 2013-14 NYSAA, 
accept the calendar or chart as evidence for that date. Score the assessment 
following the Scoring Procedures. 


Templates or other formats are provided that give or lead the student to the 
answer. For example: 


e the verifying evidence is a sequencing worksheet that contains three 
boxes that state “First,” “Next,” “Last”; the student response choices are 
pictures that contain the words “First,” “Next,” “Last.” 


e the verifying evidence is a number line on which the student must 
provide missing numbers, but the correct number is provided as a 
shaded or dotted number in the spot and the student has to put a 
sticker of the number on the spot. 


If it is determined that the evidence includes a template, record “NS” for No 
Score for baseline and/or final on which the template. Record Procedural 
Error comment 18a. 


e All of the requirements for VE are met and the additional requirements for 
photographic, digital video, or audio evidence are met. Accept what is 
documented by the teacher and score the assessment following the 
Scoring Procedures. 


e If requirements for VE and the other requirements for photographic, digital 
video, or audio evidence are not met, record “NS” for No Score for that 
date. Record Procedural Error comment 12, 14, or 15. Continue to score 
the next date. 


e All of the requirements for VE are met, the additional requirements for a 
DCS are met, and there is no obvious error in documentation. Score as 
documented on the DCS following the Scoring Procedures (all steps 
listed on the DCS are scored, unless the teacher clearly indicates 
otherwise). Direct the Scorer to record comment L. 


e lfasingle-step task is documented on a multi-step DCS, score the 
assessment following the Scoring Procedures. 


e lfasingle time segment is documented on a DCS, score the assessment 
following the Scoring Procedures. 


8a-f 


8b-f 


7or 
8b or 8d 


8c or 10 
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14 Entry includes extra Data Adjust the first DSS with date, Level of Accuracy and whether prompted from 2 
Summary Sheet(s) (i.e., separate | the second DSS (confirmed by VE) and continue to review and score the 
DSS for each piece of VE) content area. Direct the Scorer to record comment D. 

Dates 

15 Dates or information printed in Information printed in the header and/or footer of a document completed using | 1-10 
header and/or footer of the Measured Progress ProFile™ software or other web-based program (e.g., 
documents completed with News-2-You©) cannot be considered when reviewing documentation of 
Measured Progress ProFile™ or student performance data. Score the assessment following the Scoring 
other web-based program, Procedures. 
contradict information recorded on 
the evidence or VE label. 
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APPENDIX D—SUBGROUP RELIABILITY 


2013-14 NYSAA Technical Report: Appendix D—Subgroup Reliability 


Table D-1. 2013-14 NYSAA: Subgroup Reliabilities— 


English Language Arts 


Number Total Scaled Score 
elade Group Lag nie Minimum Maximum Mean siete aipber. EM 
All Students 2,687 400 525 458.07 20.89 0.81 9.13 
Male 1,859 400 525 458.78 20.91 0.81 9.22 
Female 828 400 525 456.48 20.79 0.82 8.92 
American Indian/Alaskan 
Native 20 400 525 455.10 15.88 0.78 7.41 
3 Black 677 400 525 459.22 18.15 0.76 8.92 
Asian 149 400 525 458.60 24.15 0.87 8.87 
Hispanic 721 400 525 459.27 19.42 0.78 9.06 
White 1,061 400 525 456.31 22.72 0.83 9.33 
Native Hawaiian/Other 
Pacific Islander 21 400 525 457.81 28.03 0.90 8.75 
Multi 38 400 525 463.79 21.85 0.85 8.48 
All Students 3,048 400 525 456.48 20.61 0.79 9.35 
Male 2,107 400 525 456.45 20.26 0.79 9.30 
Female 941 400 525 456.54 21.38 0.80 9.46 
American Indian/Alaskan 
Native 26 400 525 455.12 19.40 0.70 10.59 
A Black 758 400 525 457.55 19.42 0.78 9.16 
Asian 156 400 525 451.59 19.86 0.82 8.54 
Hispanic 870 400 525 457.58 20.87 0.79 9.46 
White 1,200 400 525 455.41 21.11 0.80 9.42 
Native Hawaiian/Other 
Pacific Islander 11 400 525 462.36 18.56 0.69 10.40 
Multi 27 400 525 465.89 21.33 0.79 9.79 
All Students 3,122 400 525 452.88 18.57 0.76 9.01 
Male 2,121 400 525 452.71 18.36 0.75 9.15 
Female 1,001 400 525 453.22 19.01 0.79 8.72 
American Indian/Alaskan 
Native 32 400 525 453.19 16.83 0.74 8.59 
5 Black 772 400 525 452.38 17.37 0.73 9.08 
Asian 171 400 525 454.05 20.10 0.82 8.49 
Hispanic 845 400 525 454.95 18.94 0.76 9.25 
White 1,268 400 525 451.59 18.86 0.78 8.91 
Native Hawaiian/Other 
Pacific Islander 6 400 525 
Multi 28 400 525 452.75 14.40 0.51 10.03 
All Students 3,298 400 525 456.64 20.11 0.83 8.40 
Male 2,307 400 525 456.82 19.68 0.82 8.41 
Female 991 400 525 456.23 21.08 0.84 8.38 
American Indian/Alaskan 
Native 28 400 525 454.07 14.55 0.69 8.07 
6 Black 806 400 525 457.01 19.61 0.80 8.68 
Asian 162 400 525 455.17 20.11 0.85 7.83 
Hispanic 833 400 525 458.29 19.78 0.82 8.48 
White 1,421 400 525 455.49 20.64 0.84 8.30 
Native Hawaiian/Other 
Pacific Islander 13 400 525 458.54 16.74 0.78 7.89 
continued 
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Number Total Scaled Score 
eiaee, etoue a nts Minimum Maximum Mean lta pina 22M 
6 Multi 35 400 525 463.83 18.69 0.79 8.66 
All Students 3,365 400 525 449.91 19.82 0.81 8.71 
Male 2,285 400 525 449.42 19.41 0.80 8.76 
Female 1,080 400 525 450.95 20.62 0.83 8.62 
American Indian/Alaskan 
Native 22 400 525 455.95 17.57 0.70 9.69 
7 Black 865 400 525 450.34 19.68 0.80 8.70 
Asian 175 400 525 445.79 18.58 0.80 8.36 
Hispanic 891 400 525 451.94 20.12 0.80 8.90 
White 1,373 400 525 448.72 19.82 0.81 8.66 
Native Hawaiian/Other 
Pacific Islander 8 400 525 
Multi 31 400 525 451.10 18.84 0.76 9.24 
All Students 3,167 400 525 449.24 18.62 0.82 7.79 
Male 2,166 400 525 449.31 18.88 0.83 7.84 
Female 1,001 400 525 449.11 18.05 0.82 7.68 
American Indian/Alaskan 
Native 20 400 525 447.05 18.68 0.90 5.96 
9 Black 843 400 525 450.19 19.30 0.84 7.81 
Asian 165 400 525 449.50 18.78 0.86 7.04 
Hispanic 795 400 525 450.14 18.76 0.85 7.30 
White 1,319 400 525 448.11 18.09 0.80 8.18 
Native Hawaiian/Other 
Pacific Islander 6 400 525 
Multi 19 400 525 446.47 12.33 0.56 8.19 
All Students 3,050 400 525 457.00 20.89 0.84 8.46 
Male 1,994 400 525 457.48 21.23 0.84 8.52 
Female 1,056 400 525 456.08 20.20 0.83 8.34 
American Indian/Alaskan 
Native 30 400 525 450.97 21.13 0.81 9.27 
High Black 736 400 525 458.41 20.42 0.84 8.20 
School Asian 162 400 525 454.67 21.17 0.85 8.13 
Hispanic 704 400 525 457.60 20.47 0.83 8.55 
White 1,396 400 525 456.32 21.29 0.84 8.54 
Native Hawaiian/Other 
Pacific Islander 8 400 525 
Multi 14 400 525 459.57 20.33 0.79 9.26 
Table D-2. 2013-14 NYSAA: Subgroup Reliabilities— 
Mathematics 
Number Total Scaled Score 
pigee pup See als Minimum Maximum Mean aati Ape: SEM 
All Students 2,691 400 525 457.92 21.35 0.85 8.38 
Male 1,856 400 525 458.97 21.38 0.84 8.50 
3 Female 835 400 525 455.58 21.11 0.85 8.09 
American Indian/Alaskan 
Native 20 400 525 451.65 16.88 0.77 ~8.10 
continued 
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Number Total Scaled Score 
ied? Breup St ae is Minimum Maximum Mean Sallis Alpha: “SEM 
Black 678 400 525 459.38 18.97 0.80 8.45 
Asian 149 400 525 458.04 23.09 0.86 8.69 
Hispanic 723 400 525 460.00 20.64 0.83 8.47 
3 White 1,062 400 525 455.49 22.73 0.87 8.22 
Native Hawaiian/Other Pacific 
Islander 21 400 525 459.14 28.39 0.91 8.31 
Multi 38 400 525 462.53 19.28 0.81 8.41 
All Students 3,046 400 525 454.56 22.07 0.81 9.69 
Male 2,104 400 525 455.14 21.99 0.80 9.81 
Female 942 400 525 453.27 22.19 0.82 9.44 
American Indian/Alaskan 
Native 26 400 525 457.19 22.06 0.74 11.22 
4 Black 757 400 525 455.84 21.02 0.80 9.52 
Asian 156 400 525 450.72 23.59 0.86 8.68 
Hispanic 870 400 525 456.00 22.04 0.80 9.77 
White 1,199 400 525 452.97 22.29 0.81 9.82 
Native Hawaiian/Other Pacific 
Islander 11 400 525 461.55 22.87 0.78 10.81 
Multi 27 400 525 460.07 26.24 0.82 10.99 
All Students 3,113 400 525 449.67 19.94 0.83 8.25 
Male 2,113 400 525 449.82 19.86 0.82 8.40 
Female 1,000 400 525 449.35 20.13 0.84 7.94 
American Indian/Alaskan 
Native 31 400 525 450.23 15.08 0.69 8.43 
5 Black 771 400 525 449.47 19.52 0.82 8.23 
Asian 170 400 525 450.04 21.70 0.86 7.99 
Hispanic 845 400 525 452.01 20.11 0.82 8.43 
White 1,262 400 525 448.17 19.94 0.83 8.16 
Native Hawaiian/Other Pacific 
Islander 6 400 525 
Multi 28 400 525 447.46 15.22 0.67 8.74 
All Students 3,299 400 525 453.07 21.60 0.88 7.49 
Male 2,311 400 525 453.66 21.33 0.87 7.64 
Female 988 400 525 451.69 22.18 0.90 7.17 
American Indian/Alaskan 
Native 28 400 525 449.21 19.84 0.90 6.42 
6 Black 808 400 525 453.73 21.54 0.88 7.40 
Asian 162 400 525 451.56 22.60 0.91 6.75 
Hispanic 834 400 525 454.52 21.28 0.88 7.48 
White 1,420 400 525 451.92 21.72 0.88 7.60 
Native Hawaiian/Other Pacific 400 
Islander 13 525 454.00 20.38 0.88 6.94 
Multi 34 400 525 459.50 20.63 0.77 9.87 
All Students 3,362 400 525 455.82 22.24 0.79 10.22 
Male 2,278 400 525 455.66 21.89 0.78 10.27 
Female 1,084 400 525 456.17 22.97 0.81 10.13 
7 American Indian/Alaskan 
Native 22 400 525 460.82 20.66 0.55 13.92 
Black 865 400 525 456.21 22.76 0.80 10.23 
Asian 176 400 525 454.01 22.95 0.79 10.45 
continued 
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Number Total Scaled Score 
Grade rede a. nis Minimum Maximum Mean lahie PIDNGT SOE 
Hispanic 889 400 525 458.77 22.92 0.80 10.30 
White 1,371 400 525 453.84 21.26 0.77 10.15 
7 Native Hawaiian/Other Pacific 
Islander 8 400 525 
Multi 31 400 525 454.00 19.47 0.75 9.80 
All Students 3,162 400 525 449.46 19.84 0.83 8.14 
Male 2,162 400 525 449.66 20.13 0.83 8.24 
Female 1,000 400 525 449.04 19.20 0.83 7.91 
American Indian/Alaskan Native 20 400 525 446.95 18.64 0.85 7.29 
Black 842 400 525 451.21 20.45 0.84 8.24 
8 Asian 165 400 525 448.70 20.76 0.86 7.89 
Hispanic 789 400 525 449.95 19.33 0.83 7.86 
White 1,321 400 525 448.14 19.58 0.82 8.27 
Native Hawaiian/Other Pacific 
Islander 6 400 525 
Multi 19 400 525 449.32 16.29 0.74 8.31 
All Students 3,045 400 525 458.63 22.87 0.78 10.68 
Male 1,991 400 525 459.26 22.98 0.78 10.79 
Female 1,054 400 525 457.44 22.61 0.78 10.49 
American Indian/Alaskan Native 30 400 525 452.53 20.24 0.77 9.80 
High Black 736 400 525 458.43 21.16 0.75 10.60 
Schoal Asian 163 400 525 456.29 23.02 0.82 9.81 
Hispanic 699 400 525 459.43 22.69 0.79 10.39 
White 1,396 400 525 458.71 23.89 0.79 10.98 
Native Hawaiian/Other Pacific 
Islander 8 400 525 
Multi 13 400 525 467.08 18.39 0.63 11.25 
Table D-3. 2013-14 NYSAA: Subgroup Reliabilities— 
Science 
Niumibeniak Total Scaled Score 
Grade Group Students Minimum Maximum Mean Standard HIRE SEM 
Deviation 
All Students 3,025 550 600 572.16 9.07 0.65 5.33 
Male 2,092 550 600 572.36 9.01 0.65 5.31 
Female 933 550 600 571.71 9.18 0.66 5.35 
American Indian/Alaskan 
Native 25 550 600 571.84 10.21 0.81 4.48 
4 Black 751 550 600 572.09 8.47 0.66 4.91 
Asian 155 550 600 570.83 8.86 0.72 4.66 
Hispanic 869 550 600 572.42 8.95 0.65 5.26 
White 1,187 550 600 572.08 9.49 0.64 5.72 
Native Hawaiian/Other Pacific 
Islander 11 550 600 572.82 6.79 0.67 3.89 
Multi 27 550 600 576.37 10.04 0.68 5.70 
P All Students 3,150 550 600 574.75 10.04 0.65 5.93 
Male 2,150 550 600 574.89 10.11 0.65 6.01 
continued 
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Total Scaled Score 


Number of 
Grade Group Stidents- Minin: ‘WMadnumn Mean Paes Alpha SEM 
eviation 
Female 1,000 550 600 574.45 9.90 0.66 5.74 
American Indian/Alaskan 
Native 20 550 600 575.70 9.07 0.70 4.96 
Black 837 550 600 575.17 9.96 0.62 6.13 
9 Asian 165 550 600 574.55 9.70 0.60 6.17 
Hispanic 789 550 600 575.18 9.94 0.60 6.26 
White 1,314 550 600 574.24 10.19 0.69 5.65 
Native Hawaiian/Other Pacific 
Islander 6 550 600 
Multi 19 550 600 574.32 11.25 0.88 3.82 
All Students 3,031 550 600 575.50 10.35 0.73 5.37 
Male 1,984 550 600 575.68 10.28 0.71 5.53 
Female 1,047 550 600 575.15 10.48 0.77 5.04 
American Indian/Alaskan 
Native 30 550 600 573.57 9.89 0.74 5.07 
High Black 731 550 600 576.14 9.90 0.67 5.66 
School Asian 161 550 600 573.66 10.39 0.80 4.61 
Hispanic 697 550 600 575.08 9.92 0.69 5.51 
White 1,391 550 600 575.62 10.79 0.76 5.25 
Native Hawaiian/Other Pacific 
Islander 8 550 600 
Multi 13 550 600 577.38 8.16 0.72 4.35 
Table D-4. 2013-14 NYSAA: Subgroup Reliabilities— 
Social Studies 
Number Total Scaled Score 
clade rede St oe nis Minimum Maximum Mean sahil ADRS SEM 
All Students 3,031 550 600 575.88 9.81 0.76 4.80 
Male 1,981 550 600 576.08 9.87 0.76 4.88 
Female 1,050 550 600 575.50 9.70 0.77 4.63 
American Indian/Alaskan Native 29 550 600 575.03 9.96 0.82 4.26 
High Black 731 550 600 576.19 9.73 0.75 4.84 
School Asian 160 550 600 574.84 9.92 0.73 5.12 
Hispanic 696 550 600 576.08 9.50 0.73 4.89 
White 1,394 550 600 575.72 10.02 0.78 4.71 
Native Hawaiian/Other Pacific 
Islander 8 550 600 
Multi 13 550 600 579.69 7.96 0.75 3.97 
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2013-14 NYSAA Technical Report: Appendix E—Interrater Consistency 


Table E-1. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 


English Language Arts Grade 3 


Numberor Percent ; Mean Standard 
Standard LOC Exact Responses Exact Corelation Absolute Aviation 
Matches — Scored Twice Difference 
1 162 0 100.00 1.00 0.00 
314 2 434 4 99.09 0.99 3.25 
3 30 0 100.00 1.00 0.00 
1 501 6 98.82 1.00 1.55 
322 2 102 2 98.08 0.98 3.25 
3 12 0 100.00 1.00 0.00 
1 425 8 98.15 0.99 2.71 
331 2 124 7 94.66 0.90 2.61 
3 74 0 100.00 1.00 0.00 
1 296 4 98.67 0.97 4.50 
341 2 247 3 98.80 0.99 3.37 
3 74 0 100.00 1.00 0.00 
1 389 10 97.49 1.00 0.82 0.62 

351 2 188 5 97.41 0.98 2.80 
3 41 2 95.35 0.55 5.00 


SD is blank when the number of responses scored twice is less than 10. 


Table E-2. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 


English Language Arts Grade 4 


Mean 


Standard LOC Exact Responses gies Correlation Absolute satel 
Matches Scored Twice Difference 
1 583 12 97.98 0.95 4.03 3.88 
411 2 148 0 100.00 1.00 0.00 
3 7 0 100.00 0.00 
1 430 4 99.08 0.99 2.83 
413 2 178 2 98.89 1.00 0.55 
3 106 2 98.15 0.98 2.10 
1 593 11 98.18 0.97 3.16 3.74 
432 2 77 2 97.47 0.93 4.25 
3 55 1 98.21 0.97 2.50 
1 437 7 98.42 0.99 1.89 
442 2 228 1 99.56 1.00 2.00 
3 38 0 100.00 1.00 0.00 
1 324 4 98.78 0.99 2.85 
453 2 359 8 97.82 0.99 1.48 
3 39 0 100.00 1.00 0.00 
SD is blank when the number of responses scored twice is less than 10. 
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Table E-3. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 


English Language Arts Grade 5 


Mean 


Standard LOC Exact Responses ph Correlation Absolute ee 
Matches Scored Twice Difference 

1 612 15 97.61 0.93 4.88 3.90 
512 2 100 2 98.04 0.98 2.50 
3 28 1 96.55 0.99 2.00 
ut 620 7 98.88 1.00 1.84 
523 2 131 2 98.50 1.00 0.55 
3 9 0 100.00 0.00 
1 382 6 98.45 1.00 1.35 
533 2 323 5 98.48 0.98 3.30 
3 17 0 100.00 1.00 0.00 
1 402 6 98.53 0.98 4.00 
541 2 295 5 98.33 0.99 2.46 
3 36 0 100.00 1.00 0.00 

1 459 10 97.87 0.98 3.54 3.51 
552 2 184 4 97.87 1.00 0.93 
3 72 2 97.30 0.99 1.70 


SD is blank when the number of responses scored twice is less than 10. 


Table E-4, 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 


English Language Arts Grade 6 


Percent ; Monti Standard 
Standard LOC Exact Responses Exact Corelation Absolute Deviation 
Matches _ Scored Twice Difference 
1 619 11 98.25 0.96 4.00 4.15 
611 2 164 1 99.39 1.00 2.00 
3 36 0 100.00 1.00 0.00 
1 434 7 98.41 0.97 4.50 
621 2 344 8 97.73 0.99 1.46 
3 24 0 100.00 1.00 0.00 
1 397 6 98.51 0.99 2.57 
631 2 390 7 98.24 0.95 2.54 
3 4 0 100.00 0.00 
1 615 11 98.24 0.98 2.65 2.81 
641 2 173 2 98.86 1.00 1.55 
3 15 0 100.00 1.00 0.00 
1 448 11 97.60 0.97 3.44 3.78 
651 2 247 4 98.41 0.99 1.60 
3 96 4 96.00 0.96 1.88 
SD is blank when the number of responses scored twice is less than 10. 
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Table E-5. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 


English Language Arts Grade 7 


Percent F Moan Standard 
Standard LOC Exact Responses yace Correlation Absolute Deviation 
Matches Scored Twice Difference 
1 659 14 97.92 0.99 1.44 2.76 
713 2 63 3 95.45 0.91 2.23 
3 24 0 100.00 1.00 0.00 
1 511 5 99.03 0.99 3.70 
724 2 166 4 97.65 0.97 2.40 
3 72 2 97.30 1.00 0.35 
1 615 7 98.87 1.00 1.20 
732 2 89 1 98.89 0.97 6.00 
3 14 1 93.33 0.78 6.70 
1 556 7 98.76 0.98 3.41 
741 2 154 3 98.09 0.99 1.43 
3 42 0 100.00 1.00 0.00 
1 604 10 98.37 1.00 1.26 0.96 
753 2 51 1 98.08 1.00 1.20 
3 99 1 99.00 0.99 2.00 


SD is blank when the number of responses scored twice is less than 10. 


Table E-6. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 


English Language Arts Grade 8 


Nemberol Percent : Meds Standard 
Standard LOC Exact Responses Exact Correlation Absolute Deviation 
Matches Scored Twice Difference 
1 566 15 97.42 0.97 3.01 3.07 
822 2 129 1 99.23 1.00 0.90 
3 21 0 100.00 1.00 0.00 
4 624 6 99.05 0.98 4.08 
823 2 76 2 97.44 0.87 5.25 
3 23 1 95.83 1.00 0.50 
1 637 0 100.00 1.00 0.00 
833 2 93 1 98.94 1.00 0.10 
3 20 0 100.00 1.00 0.00 
1 428 4 99.07 0.97 4.13 
842 2 232 6 97.48 0.99 1.08 
3 40 0 100.00 1.00 0.00 
1 587 17 97.19 1.00 1.19 0.78 
852 2 71 8 89.87 0.87 2.10 
3 47 2 95.92 0.90 3.00 
SD is blank when the number of responses scored twice is less than 10. 
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Table E-7. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 


English Language Arts High School 


Percent ; Mean Standard 
Standard LOC Exact Responses Eva Correlation Absolute Deviation 
Matches — Scored Twice Difference 
fl 290 9 96.99 0.99 2.28 
911 2 267 4 98.52 0.96 4.05 
3 62 1 98.41 0.99 2.50 
1 383 10 97.46 0.98 2.26 2.93 
921 2 201 2 99.01 1.00 0.10 
3 26 0 100.00 1.00 0.00 
1 341 8 97.71 0.99 1.43 
931 2 250 2 99.21 1.00 1.00 
3 41 0 100.00 1.00 0.00 
1 431 10 97.73 0.96 3.51 3.77 
942 2 117 3 97.50 0.98 2.43 
3 35 0 100.00 1.00 0.00 
1 304 6 98.06 0.99 1.53 
951 2 286 10 96.62 0.92 2.98 3.35 
3 47 2 95.92 0.96 2.55 


SD is blank when the number of responses scored twice is less than 10. 


Table E-8. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 


Mathematics Grade 3 


Mean 


Standard LOC Exact Responses sae Correlation Absolute ee 
Matches Scored Twice Difference 

1 512 5 99.03 0.98 4.80 

301 2 69 0 100.00 1.00 0.00 

3 57 0 100.00 1.00 0.00 

1 259 6 97.74 0.99 2.45 

302 2 290 1 99.66 1.00 2.00 

3 56 5 91.80 0.80 2.34 

1 467 6 98.73 1.00 1.03 

303 2 138 5 96.50 0.97 2.40 

3 13 0 100.00 1.00 0.00 

1 200 4 98.04 0.98 3.85 

304 2 377 7 98.18 0.99 2.63 

3 13 0 100.00 1.00 0.00 

1 223 6 97.38 0.97 3.55 

305 2 305 8 97.44 0.96 2.49 

3 75 1 98.68 1.00 0.20 

SD is blank when the number of responses scored twice is less than 10. 
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Table E-9. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 


Mathematics Grade 4 


Percent F Mean Standard 
Standard LOC Exact Responses Exact Correlation Absolute Deviation 
Matches Scored Twice Difference 
1 524 10 98.13 0.97 3.52 3.13 
401 2 152 4 97.44 1.00 0.60 
3 53 0 100.00 1.00 0.00 
ut 530 16 97.07 0.98 2.96 2.23 
402 2 71 2 97.26 1.00 1.00 
3 134 2 98.53 0.98 2.50 
1 578 13 97.80 0.98 2.56 2.78 
403 2 146 3 97.99 0.98 2.33 
3 5 0 100.00 0.00 
1 559 7 98.76 0.98 2.94 
404 2 148 3 98.01 0.98 1.73 
3 20 2 90.91 0.95 1.25 
1 415 9 97.88 0.96 2.92 
405 2 158 3 98.14 0.99 2.10 
3 151 AE 99.34 0.99 4.00 


SD is blank when the number of responses scored twice is less than 10. 


Table E-10. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 


Mathematics Grade 5 


Nemberor Percent : Meds Standard 
Standard LOC Exact Responses Exact Correlation Absolute Devishon 
Matches Scored Twice Difference 

1 514 10 98.09 0.98 3.66 3.74 
501 2 108 8 93.10 0.88 4.34 

3 100 1 99.01 0.99 2.50 

4 619 16 97.48 0.98 2.59 3.25 
502 2 87 5 94.57 0.98 1.44 

3 37 1 97.37 0.99 1.30 

1 601 14 97.72 0.98 3.21 3.15 
503 2 81 4 95.29 0.93 3.15 

3 6 1 85.71 3.30 

1 647 14 97.88 0.98 2.21 2.19 
504 2 64 2 96.97 0.99 1.45 

3 39 1 97.50 0.99 2.00 

1 415 10 97.65 0.99 1.91 1.30 
505 2 296 2 99.33 1.00 1.10 

3 22 0 100.00 1.00 0.00 

SD is blank when the number of responses scored twice is less than 10. 
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Table E-11. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 
Mathematics Grade 6 


Numba, Percent , Mean Standard 
Standard LOC Exact Responses Exact Correlation Absolute Deviation 
Matches — Scored Twice Difference 
1 661 12 98.22 0.98 2.39 2.85 
605 2 125 4 96.90 1.00 0.45 
3 22 0 100.00 1.00 0.00 
1 489 3 99.39 0.99 3.97 
606 2 248 7 97.25 0.93 3.79 
3 63 1 98.44 1.00 0.10 
1 557 12 97.89 1.00 1.53 1.29 
607 2 237 5 97.93 0.88 5.46 
3 15 0 100.00 1.00 0.00 
1 457 18 96.21 0.95 3.59 2.47 
608 2 320 4 98.77 0.97 3.38 
3 13 0 100.00 1.00 0.00 
ut 600 11 98.20 0.97 3.84 4.17 
618 2 120 7 94.49 0.97 2.10 
3 76 1 98.70 1.00 0.30 


SD is blank when the number of responses scored twice is less than 10. 


Table E-12. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 
Mathematics Grade 7 


fem 11/1 Percent : Mean Standard 
Standard LOC Exact Responses Pyact Correlation Absolute Deviation 
Matches Scored Twice Difference 
1 275 8 97.17 0.98 2.33 
705 2 408 10 97.61 0.99 1.66 1.88 
3 42 0 100.00 1.00 0.00 
1 409 12 97.15 0.96 4.57 3.71 
706 2 240 7 97.17 0.98 1.97 
3 74 5 93.67 0.94 2.06 
1 608 20 96.82 0.98 2.77 3.05 
707 2 102 4 96.23 0.99 1.85 
3 31 0 100.00 1.00 0.00 
1 553 9 98.40 0.98 2.66 
708 2 67 4 94.37 0.85 4.13 
3 122 3 97.60 0.92 3.50 
1 444 8 98.23 1.00 0.91 
710 2 226 2 99.12 1.00 1.85 
3 81 3 96.43 1.00 0.93 


SD is blank when the number of responses scored twice is less than 10. 
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Table E-13. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 


Mathematics Grade 8 


Percent F Moan Standard 
Standard LOC Exact Responses Eyare Correlation Absolute Deviation 
Matches Scored Twice Difference 
1 495 8 98.41 0.97 2.95 
805 2 209 2 99.05 0.94 5.05 
3 28 2 93.33 0.97 2.55 
1 566 5 99.12 1.00 1.08 
808 2 123 2 98.40 1.00 0.90 
3 41 1 97.62 0.93 5.00 
1 557 12 97.89 0.98 2.56 2.79 
809 2 146 3 97.99 0.99 1.20 
3 17 0 100.00 1.00 0.00 
1 487 7 98.58 0.98 2.73 
810 2 128 2 98.46 0.99 1.45 
3 105 1 99.06 0.95 10.00 
1 683 25 96.47 0.98 2.04 2.03 
818 2 25 0 100.00 1.00 0.00 
3 14 0 100.00 1.00 0.00 


SD is blank when the number of responses scored twice is less than 10. 


Table E-14. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 


Mathematics High School 


NEMDeEO! Percent : Meds Standard 
Standard LOC Exact Responses Eyant Correlation Absolute Deviation 
Matches Scored Twice Difference 
1 357 7 98.08 0.99 2.37 
911 2 144 1 99.31 0.94 8.00 
3 144 4 97.30 0.99 2.25 
4 269 8 97.11 0.97 3.83 
912 2 337 5 98.54 0.98 3.02 
3 21 0 100.00 1.00 0.00 
1 440 12 97.35 0.99 1.75 1.68 
913 2 169 HE 99.41 1.00 2.50 
3 26 0 100.00 1.00 0.00 
1 367 9 97.61 0.99 2.68 
914 2 132 8 94.29 0.93 3.51 
3 110 4 96.49 0.93 4.45 
1 336 8 97.67 0.97 3.94 
915 2 149 2 98.68 1.00 1.05 
3 120 3 97.56 0.99 2.07 
SD is blank when the number of responses scored twice is less than 10. 
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Table E-15. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 
Science Grade 4 


Number of Mean 


Standard LOC Exact Responses patel Correlation Absolute aia 
Matches _ Scored Twice Difference 
1 468 10 97.91 0.97 2.98 3.12 
411 2 242 3 98.78 1.00 1.30 
3 19 1 95.00 0.93 2.00 
1 567 6 98.95 0.97 4.47 
422 2 131 3 97.76 0.99 2.17 
3 43 1 97.73 0.96 2.50 


SD is blank when the number of responses scored twice is less than 10. 


Table E-16. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 
Science Grade 8 


ee IE OL — Percent : Meal Standard 
Standard LOC Exact Responses Exact Correlation Absolute Deviation 
Matches Scored Twice Difference 
1 401 0 100.00 1.00 0.00 
813 2 264 3 98.88 0.99 2.27 
3 59 2 96.72 0.60 5.15 
1 331 5 98.51 0.96 3.96 
832 2 323 9 97.29 0.98 1.06 
3 67 2 97.10 0.99 1.40 


SD is blank when the number of responses scored twice is less than 10. 


Table E-17. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 
Science High School 


Number of Mean 


Standard LOC Exact Responses pb Correlation Absolute aa 
Matches — Scored Twice Difference 
1 328 8 97.62 0.98 2.38 
921 2 211 2 99.06 0.99 3.00 
3 100 2 98.04 0.88 5.50 
1 260 2 99.24 0.99 2.60 
931 2 324 7 97.89 0.99 1.61 
3 52 0 100.00 1.00 0.00 


SD is blank when the number of responses scored twice is less than 10. 
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Table E-18. 2013-14 NYSAA: Interrater Consistency Statistics for SILOC Items — 
Social Studies High School 


Number of Mean 


Standard LOC Exact Responses Gabel Correlation Absolute eat 
Matches — Scored Twice Difference 
1 241 6 97.57 0.99 1.42 
911 2 384 4 98.97 0.97 4.38 
3 16 al 94.12 1.00 0.50 
1 249 5 98.03 0.97 6.00 
921 2 357 5 98.62 0.99 1.94 
3 39 1 97.50 1.00 0.50 


SD is blank when the number of responses scored twice is less than 10. 
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Table F-1. 2013-14 NYSAA: Performance Level Distributions 
by Content Area and Grade 


Performance’ Percent at Level 


Content Area Grade 


Level 2013-14 
21 6.51 
3 22 23.37 
23 64.20 
24 5.92 
21 4.92 
A 22 19.91 
23 69.29 
24 5.87 
21 5.73 
5 22 21.04 
23 68.71 
24 4.52 
English Ss ea 
ae 6 - 20.13 
Arts 23 69.10 
24 4.61 
21 11.41 
7 22 20.68 
23 63.12 
24 4.78 
21 12.72 
8 22 17.30 
23 64.32 
24 5.65 
21 7.48 
High 22 25.08 
School 23 62.23 
24 5.21 
21 11.26 
3 22 20.33 
23 63.25 
24 5.17 
21 10.24 
A 22 20.78 
23 62.25 
24 6.73 
Mathematics ae pee 
5 22 20.01 
23 66.27 
24 5.17 
21 7.40 
6 22 23.98 
23 60.08 
24 8.55 
7 21 7.85 
22 22.52 
continued 
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Content Area Grade 


Performance Percent at Level 


Level 2013-14 
7 23 63.30 
24 6.34 
21 10.34 
8 22 21.66 
Mathematics 23 63.44 
24 4.55 
21 7.22 
High 22 19.38 
School 23 68.47 
24 4.93 
21 5.19 
A 22 16.50 
23 73.02 
24 5.29 
21 4.73 
Science 8 22 21.94 
23 65.65 
24 7.68 
21 4.49 
High 22 20.72 
School 23 65.98 
24 8.81 
21 5.58 
Social Studies pee . oe 
24 5.67 
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APPENDIX G—CUMULATIVE DISTRIBUTION GRAPHS 
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Figure G-1. 2013-14 NYSAA: Cumulative Score Distributions 


Top: English Language Arts Grade 3. Bottom: English Language Arts Grade 4 


Cumulative Scale Score Distributions: English Language Arts Grade 3 
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Cumulative Scale Score Distributions: English Language Arts Grade 4 
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— 2014 


Figure G-2. 2013-14 NYSAA: Cumulative Score Distributions 
Top: English Language Arts Grade 5 Bottom: English Language Arts Grade 6 


Cumulative Scale Score Distributions: English Language Arts Grade 5 


15 — 2014 
0.8- 


0.6 


0.4 


Cumulative Proportion 


0.2 


400 425 450 475 500 525 


Scale Score 


Cumulative Scale Score Distributions: English Language Arts Grade 6 
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Figure G-3. 2013-14 NYSAA: Cumulative Score Distributions 


Top: English Language Arts Grade 7 Bottom: English Language Arts Grade 8 


Cumulative Scale Score Distributions: English Language Arts Grade 7 
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Cumulative Scale Score Distributions: English Language Arts Grade 8 
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Figure G-4, 2013-14 NYSAA: Cumulative Score Distributions 
Top: English Language Arts Grade 11 Bottom: Mathematics Grade 3 


Cumulative Scale Score Distributions: English Language Arts Grade HS 
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Cumulative Scale Score Distributions: Mathematics Grade 3 
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Figure G-5. 2013-14 NYSAA: Cumulative Score Distributions 
Top: Mathematics Grade 4 Bottom: Mathematics Grade 5 


Cumulative Scale Score Distributions: Mathematics Grade 4 
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Cumulative Scale Score Distributions: Mathematics Grade 5 
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Figure G-6. 2013-14 NYSAA: Cumulative Score Distributions 
Top: Mathematics Grade 6 Bottom: Mathematics Grade 7 


Cumulative Scale Score Distributions: Mathematics Grade 6 
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Cumulative Scale Score Distributions: Mathematics Grade 7 
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Figure G-7. 2013-14 NYSAA: Cumulative Score Distributions 
Top: Mathematics Grade 8 Bottom: Mathematics Grade 11 
Cumulative Scale Score Distributions: Mathematics Grade 8 
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Cumulative Scale Score Distributions: Mathematics Grade HS 
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Figure G-8. 2013-14 NYSAA: Cumulative Score Distributions 
Top: Science Grade 4 Bottom: Science Grade 8 


Cumulative Scale Score Distributions: Science Grade 4 
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Figure G-9. 2013-14 NYSAA: Cumulative Score Distributions 
Top: Science High School Bottom: Social Studies High School 


Cumulative Scale Score Distributions: Science Grade HS 
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Cumulative Scale Score Distributions: Social Studies Grade HS 
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APPENDIX H—CLASSICAL ITEM ANALYSIS 
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Table H-1. 2013-14 NYSAA: Classical Test Theory Statistics for S/LOC Items 


English Language Arts Grade 3 


Standard LOC sept N Mean SD Discrimination _ Difficulty 
1 100 763 81.11 29.80 0.56 0.81 
314 2 100 1,755 85.66 25.57 0.46 0.86 
3 100 122 88.62 21.62 0.28 0.89 
1 100 2,172 78.99 30.62 0.54 0.79 
322 2 100 383 86.03 23.01 0.30 0.86 
3 100 56 93.48 18.05 0.43 0.93 
di 100 1811 82.54 26.26 0.58 0.83 
331 2 100 552 89.64 16.86 0.31 0.90 
3 100 290 92.77. 17.34 0.07 0.93 
1 100 1,362 78.60 29.61 0.59 0.79 
341 2 100 1,012 86.87 24.94 0.32 0.87 
3 100 262 89.12 21.96 0.35 0.89 
i 100 1,642. 77.86 27.55 0.59 0.78 
351 2 100 832 84.68 26.54 0.34 0.85 
3 100 182 90.32 22.32 0.16 0.90 


Table H-2. 2013-14 NYSAA: Classical Test Theory Statistics for S/LOC Items 


English Language Arts Grade 4 


Standard Loc ™ eee Mean SD Discrimination _ Difficulty 
1 100 2,315 85.41 25.34 0.51 0.85 
411 2 100 652 89.78 18.13 0.25 0.90 
3 100 46 95.80 11.58 0.34 0.96 
1 100 1,723 83.72 28.49 0.60 0.84 
413 2 100 694 89.82 20.62 0.43 0.90 
3 100 497 88.86 17.82 0.29 0.89 
HE 100 2,413 76.08 28.60 0.50 0.76 
432 2 100 327 88.98 19.18 0.24 0.89 
3 100 244 92.19 16.92 0.33 0.92 
1 100 1,807 83.95 23.90 0.46 0.84 
442 2 100 927 88.89 19.30 0.34 0.89 
3 100 179 91.44 13.84 0.30 0.91 
1 100 1,302 80.52 29.81 0.61 0.81 
453 2 100 1,473 87.34 19.32 0.25 0.87 
3 100 207 91.61 16.11 0.29 0.92 
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Table H-3. 2013-14 NYSAA: Classical Test Theory Statistics for S/LOC Items 


English Language Arts Grade 5 


Standard LOC ween N Mean SD Discrimination _ Difficulty 
1 100 2,503 81.68 25.51 0.49 0.82 
512 2 100 433 85.74. 22.21 0.36 0.86 
3 100 103 93.03 13.08 0.35 0.93 
‘i 100 2,539 82.38 26.82 0.52 0.82 
523 2 100 493 86.59 19.24 0.36 0.87 
3 100 29 94.66 14.36 0.48 0.95 
i 100 1,663 80.89 29.09 0.54 0.81 
533 2 100 1,300 85.43 25.19 0.46 0.85 
3 100 58 95.79 8.94 0.26 0.96 
1 100 1,719 84.77 2741 0.52 0.85 
541 2 100 1,174 84.05 22.04 0.48 0.84 
3 100 132 90.31 (21.18 0.41 0.90 
i 100 1,908 75.77 30.89 0.50 0.76 
552 2 100 779 84.73 26.48 0.37 0.85 
3 100 321 82.52 28.14 0.44 0.83 


Table H-4. 2013-14 NYSAA: Classical Test Theory Statistics for S/LOC Items 


English Language Arts Grade 6 


Standard Loc ™ Sore oN Mean SD Discrimination _Difficulty 
1 100 2,505 83.31 28.74 0.51 0.83 
611 2 100 612 90.27 19.93 0.38 0.90 
3 100 149 89.89 16.19 0.27 0.90 
di 100 1,765 82.62 29.45 0.55 0.83 
621 2 100 1,378 88.51 20.27 0.31 0.89 
3 100 95 93.44 15.33 0.42 0.93 
1 100 1,645 82.48 26.82 0.60 0.82 
631 2 100 1,549 88.46 18.43 0.37 0.88 
3 100 20 95.15 8.88 0.43 0.95 
1 100 2,387 83.96 25.14 0.56 0.84 
641 2 100 744 89.23 17.61 0.28 0.89 
3 100 62 88.40 19.44 0.26 0.88 
ut 100 1,779 79.18 29.21 0.58 0.79 
651 2 100 1,064 87.08 17.91 0.29 0.87 
3 100 393 89.30 17.98 0.24 0.89 
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Table H-5. 2013-14 NYSAA: Classical Test Theory Statistics for S/LOC Items 
English Language Arts 


Maximum 


Standard LOC See ie N Mean SD Discrimination Difficulty 
1 100 2,839 83.58 27.69 0.50 0.84 
713 2 100 314 91.05 18.49 0.21 0.91 
3 100 118 88.34 20.38 0.46 0.88 
1 100 2,272 80.16 29.30 0.59 0.80 
724 2 100 695 87.79 20.86 0.27 0.88 
3 100 306 86.66 23.48 0.50 0.87 
1 100 2,715 82.16 28.86 0.49 0.82 
732 2 100 410 86.05 21.69 0.22 0.86 
3 100 55 86.29 27.08 0.53 0.86 
1 100 2,351 83.98 29.01 0.50 0.84 
741 2 100 703 88.15 20.60 0.19 0.88 
3 100 168 86.99 23.78 0.42 0.87 
1 100 2,651 80.87 26.22 0.49 0.81 
753 2 100 212 87.40 17.70 0.22 0.87 
3 100 422 88.79 15.90 0.17 0.89 


Table H-6. 2013-14 NYSAA: Classical Test Theory Statistics for S/LOC Items 


Maximum 


English Language Arts Grade 8 


Standard LOC SaGro N Mean SD Discrimination Difficulty 
1 100 2,477 79.09 29.44 0.52 0.79 
822 2 100 480 84.69 26.19 0.46 0.85 
3 100 75 95.63 12.48 0.37 0.96 
1 100 2,651 84.91 27.09 0.51 0.85 
823 2 100 334 83.44 25.57 0.29 0.83 
3 100 106 89.88 18.85 0.39 0.90 
1 100 2,636 83.50 29.02 0.51 0.83 
833 2 100 375 87.37 23.31 0.33 0.87 
3 100 92 92.73 17.00 0.40 0.93 
HE 100 1,855 82.94 23.53 0.56 0.83 
842 2 100 1,014 87.27 17.94 0.25 0.87 
3 100 177 90.98 16.61 0.44 0.91 
a 100 2,580 80.54 25.82 0.50 0.81 
852 2 100 329 85.67 18.92 0.24 0.86 
3 100 204 90.55 15.76 0.35 0.91 
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Table H-7. 2013-14 NYSAA: Classical Test Theory Statistics for S/LOC Items 
English Language Arts High School 


Maximum 


Standard LOC Score N Mean SD Discrimination Difficulty 
1 100 1,437 77.75 31.89 0.56 0.78 
911 2 100 1,256 84.33 24.25 0.31 0.84 
3 100 270 85.07 18.22 0.34 0.85 
1 100 1,814 80.42 27.95 0.56 0.80 
921 2 100 1,007 86.06 24.90 0.24 0.86 
3 100 87 87.28 23.19 0.52 0.87 
1 100 1,657 82.94 25.43 0.58 0.83 
931 2 100 1,159 90.93 17.29 0.23 0.91 
3 100 178 87.40 17.87 0.15 0.87 
1 100 2,209 82.51 26.82 0.51 0.83 
942 2 100 575 83.93 22.51 0.25 0.84 
3 100 121 88.23 16.24 0.32 0.88 
1 100 1,426 74.84 30.33 0.61 0.75 
951 2 100 1,385 81.69 20.69 0.27 0.82 
3 100 202 79.18 25.14 0.38 0.79 


Table H-8. 2013-14 NYSAA: Classical Test Theory Statistics for S/LOC Items 
Mathematics Grade 3 


Maximum 


Standard LOC Gaara N Mean SD Discrimination Difficulty 
1 100 2,219 79.01 28.63 0.65 0.79 
301 2 100 244 82.12 24.32 0.50 0.82 
3 100 203 87.16 23.51 0.47 0.87 
1 100 1,185 76.44 29.29 0.61 0.76 
302 2 100 1,273 84.99 21.56 0.44 0.85 
3 100 192 91.41 19.33 0.16 0.91 
4 100 2,064 82.99 27.56 0.57 0.83 
303 2 100 505 84.81 24.02 0.49 0.85 
3 100 52 92.12 16.92 0.28 0.92 
HE 100 964 73.42 34.10 0.55 0.73 
304 2 100 1,623 83.83 24.00 0.46 0.84 
3 100 48 92.00 16.29 0.92 
ut 100 1,031 75.21 31.29 0.65 0.75 
305 2 100 1,338 87.68 21.12 0.56 0.88 
3 100 269 90.80 18.29 0.35 0.91 
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Table H-9. 2013-14 NYSAA: Classical Test Theory Statistics for S/LOC Items 
Mathematics Grade 4 


Maximum 


Standard LOC See ie N Mean SD Discrimination Difficulty 
1 100 2,102 80.81 28.11 0.52 0.81 
401 2 100 617 85.06 23.42 0.33 0.85 
3 100 269 89.12 20.24 0.41 0.89 
1 100 2,157 73.99 30.91 0.58 0.74 
402 2 100 240 83.03 21.62 0.35 0.83 
3 100 592 84.35 21.39 0.45 0.84 
1 100 2,298 80.78 28.67 0.57 0.81 
403 2 100 634 88.57 21.42 0.22 0.89 
3 100 31 84.77 28.45 0.35 0.85 
1 100 2,301 83.48 26.41 0.55 0.83 
404 2 100 575 85.55 20.94 0.39 0.86 
3 100 114 89.68 16.11 0.27 0.90 
1 100 1,751 84.17 25.52 0.59 0.84 
405 2 100 617 88.08 21.15 0.40 0.88 
3 100 610 87.61 23.48 0.25 0.88 


Table H-10. 2013-14 NYSAA: Classical Test Theory Statistics for S/LOC Items 
Mathematics Grade 5 


Standard Loc ™ pt N Mean SD Discrimination _ Difficulty 
1 100 2,128 78.13 30.19 0.65 0.78 
501 2 100 519 80.67 30.54 0.28 0.81 
3 100 361 89.98 18.10 0.20 0.90 
1 100 2,492 76.70 29.39 0.58 O27 
502 2 100 434 84.18 22.97 0.33 0.84 
3 100 129 85.34 20.54 0.33 0.85 
1 100 2,562 79.06 30.84 0.59 0.79 
503 2 100 330 83.95 27.40 0.40 0.84 
3 100 50 90.10 23.54 0.23 0.90 
HE 100 2,609 83.21 25.72 0.59 0.83 
504 2 100 298 86.30 23.35 0.32 0.86 
3 100 152 84.63 25.28 0.35 0.85 
1 100 1,715 81.64 28.31 0.66 0.82 
505 2 100 1,228 90.20 17.56 0.36 0.90 
3 100 86 91.86 16.59 0.32 0.92 
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Table H-11. 2013-14 NYSAA: Classical Test Theory Statistics for S/LOC Items 
Mathematics Grade 6 


Maximum 


Standard LOC Shore N Mean SD Discrimination Difficulty 
1 100 2,633 82.29 27.24 0.59 0.82 
605 2 100 517 87.47 22.46 0.33 0.87 
3 100 86 96.35 9.05 0.04 0.96 
1 100 1,910 81.45 27.72 0.64 0.81 
606 2 100 1,044 89.46 20.66 0.32 0.89 
3 100 255 89.21 18.76 0.18 0.89 
1 100 2,220 79.56 29.80 0.63 0.80 
607 2 100 959 86.04 22.29 0.27 0.86 
3 100 50 92.68 14.86 0.34 0.93 
1 100 1,866 79.91 28.36 0.69 0.80 
608 2 100 1,289 84.50 21.55 0.27 0.84 
3 100 63 85.10 23.67 0.46 0.85 
1 100 2,415 76.94 29.52 0.61 0.77 
618 2 100 499 85.78 23.06 0.42 0.86 
3 100 306 84.51 21.90 0.36 0.85 


Table H-12. 2013-14 NYSAA: Classical Test Theory Statistics for SILOC Items 
Mathematics Grade 7 


Standard Loc ™ Se N Mean SD Discrimination _ Difficulty 
1 100 1,221 69.41 37.33 0.51 0.69 
705 2 100 1,855 85.11 24.96 0.46 0.85 
3 100 188 95.38 12.10 0.27 0.95 
1 100 1,803 76.21 31.92 0.59 0.76 
706 2 100 1,085 82.77 25.24 0.49 0.83 
3 100 328 86.86 23.58 0.49 0.87 
1 100 2,692 76.67 32.12 0.56 0.77 
707 2 100 467 76.66 32.84 0.48 0.77 
3 100 113 89.87 20.10 0.39 0.90 
ME 100 2,401 79.34 28.73 0.58 0.79 
708 2 100 301 83.64 25.25 0.47 0.84 
3 100 555 86.71 21.48 0.36 0.87 
ut 100 1,902 81.00 27.48 0.59 0.81 
710 2 100 1,046 87.11 20.67 0.32 0.87 
3 100 317 88.74 24.68 0.57 0.89 
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Table H-13. 2013-14 NYSAA: Classical Test Theory Statistics for SILOC Items 
Mathematics Grade 8 


Maximum 


Standard LOC See ie N Mean SD Discrimination Difficulty 
1 100 2,133 83.82 25.50 0.57 0.84 
805 2 100 832 88.38 20.21 0.38 0.88 
3 100 129 84.53 30.81 0.41 0.85 
1 100 2,454 82.10 30.79 0.61 0.82 
808 2 100 481 84.38 29.10 0.39 0.84 
3 100 166 89.64 18.75 0.33 0.90 
1 100 2,420 76.20 31.35 0.63 0.76 
809 2 100 595 89.12 20.65 0.34 0.89 
3 100 66 80.92 27.77 0.53 0.81 
1 100 2,043 81.49 29.81 0.64 0.81 
810 2 100 555 87.88 21.69 0.36 0.88 
3 100 431 82.55 31.70 0.36 0.83 
1 100 2,979 77.33 29.81 0.54 0.77 
818 2 100 90 90.76 21.09 0.36 0.91 
3 100 45 89.91 18.83 0.18 0.90 


Table H-14,. 2013-14 NYSAA: Classical Test Theory Statistics for SILOC Items 
Mathematics High School 


Standad Loc ™ per N Mean SD Discrimination _ Difficulty 
1 100 1,707 80.53 27.43 0.57 0.81 
911 2 100 634 86.62 19.66 0.27 0.87 
3 100 638 81.01 26.29 0.33 0.81 
1 100 1,311 72.51 34.21 0.53 0.73 
912 2 100 1,541 78.42 27.09 0.41 0.78 
3 100 95 81.94 24.55 0.41 0.82 
1 100 2,080 77.23 31.11 0.55 0.77 
913 2 100 767 78.80 30.46 0.44 0.79 
3 100 146 85.17 25.57 0.40 0.85 
A 100 1,780 79.36 29.88 0.56 0.79 
914 2 100 627 82.29 24.73 0.40 0.82 
3 100 538 85.40 24.84 0.22 0.85 
1 100 1,615 77.59 32.69 0.55 0.78 
915 2 100 697 87.02 24.35 0.29 0.87 
3 100 591 81.46 25.49 0.38 0.81 
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Table H-15. 2013-14 NYSAA: Classical Test Theory Statistics for S/LOC Items 
Science Grade 4 
Maximum 


Standard LOC Shore N Mean SD Discrimination Difficulty 
1 100 1,822 83.28 26.92 0.45 0.83 
411 2 100 981 87.63 22.44 0.22 0.88 
3 100 141 95.35 10.88 0.16 0.95 
1 100 2,221 86.89 23.94 0.39 0.87 
422 2 100 561 87.78 23.79 0.27 0.88 
3 100 193 91.23 16.88 0.30 0.91 


Table H-16. 2013-14 NYSAA: Classical Test Theory Statistics for SILOC Items 
Science Grade 8 


Maximum 


Standard LOC Score N Mean SD Discrimination Difficulty 
1 100 1,665 80.94 28.27 0.40 0.81 
813 2 100 1,154 86.95 23.67 0.33 0.87 
3 100 239 93.18 18.02 0.28 0.93 
1 100 1,431 82.37 26.01 0.42 0.82 
832 2 100 1,357 88.14 17.64 0.24 0.88 
3 100 289 89.02 18.12 0.31 0.89 


Table H-17. 2013-14 NYSAA: Classical Test Theory Statistics for S/LOC Items 
Science High School 
Maximum 


Standard LOC SrGhe N Mean SD Discrimination Difficulty 
1 100 1,541 82.62 26.64 0.45 0.83 
921 2 100 1,040 85.13 22.83 0.23 0.85 
3 100 400 88.66 20.98 0.15 0.89 
1 100 1,276 81.46 27.97 0.45 0.81 
931 2 100 1,493 85.99 24.13 0.28 0.86 
3 100 193 92.16 15.70 0.08 0.92 


Table H-18. 2013-14 NYSAA: Classical Test Theory Statistics for SILOC Items 
Social Studies High School 


Standard LOC eee N Mean SD Discrimination _ Difficulty 
1 100 1,136 79.49 29.30 0.52 0.79 
911 2 100 1,761 87.01 23.43 0.28 0.87 
3 100 83 82.81 21.87 0.17 0.83 
1 100 1,106 73.11 36.74 0.48 0.73 
921 2 100 1,735 86.97 23.51 0.37 0.87 
3 100 142 93.08 16.66 0.12 0.93 
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Table H-19. 2013-14 NYSAA: Classical Test Theory Statistics for Standards-Based Items 


Subject Grade Standard en N Mean SD Difficulty Discrimination 
314 25 2,640 14.13 4.95 0.57 0.50 
322 25 2,611 9.46 4.68 0.38 0.62 
3 331 25 2,653 11.71 5.98 0.47 0.61 
341 25 2,636 12.65 6.06 0.51 0.60 
351 25 2,656 11.46 5.73 0.46 0.61 
411 25 3,013 10.50 4.40 0.42 0.62 
413 25 2,914 12.95 6.49 0.52 0.56 
4 432 25 2,984 9.93 5.74 0.40 0.49 
442 25 2,913 11.91 5.30 0.48 0.56 
453 25 2,982 13.21 5.51 0.53 0.57 
512 25 3,039 9.84 4.59 0.39 0.54 
523 25 3,061 9.67 4.14 0.39 0.59 
5 533 25 3,021 11.83 5.09 0.47 0.54 
541 25 3,025 12.04 5.06 0.48 0.49 
552 25 3,008 11.42 6.20 0.46 0.47 
611 25 3,266 10.58 5.09 0.42 0.60 
English 621 25 3,238 12.18 5.16 0.49 0.65 
Language 6 631 25 3,214 12.25 4.75 0.49 0.59 
Arts 641 25 3,193 10.57 4.52 0.42 0.59 
651 25 3,236 12.59 6.17 0.50 0.62 
713 25 3,271 9.71 4.53 0.39 0.59 
724 25 3,273 11.23 5.87 0.45 0.58 
7 732 25 3,180 9.50 4.28 0.38 0.58 
741 25 3,222 10.92 5.18 0.44 0.56 
753 25 3,285 10.64 5.99 0.43 0.57 
822 25 3,032 9.60 4.74 0.38 0.61 
823 25 3,091 9.82 4.41 0.39 0.64 
8 833 25 3,103 9.78 4.57 0.39 0.63 
842 25 3,046 11.85 5.24 0.47 0.55 
852 25 3,113 9.95 5.10 0.40 0.57 
911 25 2,963 12.67 5.88 0.51 0.57 
921 25 2,908 11.30 5.14 0.45 0.60 
High School 931 25 2,994 12.42 5.36 0.50 0.64 
942 25 2,905 10.41 4.86 0.42 0.57 
951 25 3,013 12.28 5.52 0.49 0.68 
301 25 2,666 9.82 5.36 0.39 0.64 
Mathematics 3 302 25 2,650 12.85 5.70 0.51 0.72 
303 25 2,621 10.09 4.51 0.40 0.65 
continued 
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Subject Grade Item ees N Mean SD Difficulty Discrimination 
3 304 25 2,635 12.91 5,19 0.52 0.63 
305 25 2,638 13.65 5.97 0.55 0.60 
401 25 2,988 11.14 5.76 0.45 0.64 
402 25 2,989 11.25 7.07 0.45 0.54 
4 403 25 2,963 10.01 4.57 0.40 0.63 
404 25 2,990 10.43 4.77 0.42 0.62 
405 25 2,978 13.20 6.65 0.53 0.54 
501 25 3,008 11.09 6.29 0.44 0.61 
502 25 3,055 9.51 4.99 0.38 0.64 
5 503 25 2,942 9.08 4.41 0.36 0.65 
504 25 3,059 9.83 4.64 0.39 0.61 
505 25 3,029 12.01 5.17 0.48 0.56 
605 25 3,236 9.95 4.62 0.40 0.68 
606 25 3,209 12.10 5.73 0.48 0.71 
6 607 25 3,229 10.63 4.93 0.43 0.71 
608 25 3,218 11.48 4.95 0.46 0.73 
Mathematics 618 25 3,220 10.49 5.87 0.42 0.69 
705 25 3,264 13.11 5.95 0.52 0.47 
706 25 3,216 12.01 6.16 0.48 0.58 
7 707 25 3,272 9.30 4.97 0.37 0.58 
708 25 3,257 11.35 6.62 0.45 0.58 
710 25 3,265 12.23 5.88 0.49 0.57 
805 25 3,094 11.15 4.98 0.45 0.62 
808 25 3,101 10.25 5,23 0.41 0.62 
8 809 25 3,081 9.65 4.96 0.39 0.68 
810 25 3,029 11.79 6.30 0.47 0.59 
818 25 3,114 8.22 3.81 0.33 0.60 
911 25 2,979 13.00 6.68 0.52 0.61 
912 25 2,947 12.00 5.40 0.48 0.49 
High School 913 25 2,993 10.46 5.40 0.42 0.43 
914 25 2,945 12.45 6.70 0.50 0.61 
915 25 2,903 12.92 6.88 0.52 0.54 
A 411 25 2,944 11.75 5.30 0.47 0.46 
422 25 2,975 11.12 5.09 0.44 0.44 
Scarce 8 813 25 3,058 12.42 5.78 0.50 0.45 
832 25 3,077 13.27 5.60 0.53 0.45 
High School 921 25 2,981 13.06 6.06 0.52 0.53 
931 25 2,962 13.20 5.44 0.53 0.54 
911 25 2,980 13.25 5.03 0.53 0.60 
papal eludes. HIG ecnoo! 921 25 2,983 13.29 5.68 0.53 0.57 
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APPENDIX I—CORRELATIONS BETWEEN STANDARDS-BASED 
ITEMS 
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Table I-1. 2013-14 NYSAA: Standards-Based Item Correlations for Grade 3 
Subject ee N Correlation 

314 322 2,574 0.40 

314 331 2,610 0.36 

314 341 2,596 0.43 

314 351 2,612 0.41 


ee 322 331 2,586 0.51 
hades 322 341 2,570 0.51 


322 351 2,586 0.52 
331 341 2,608 0.50 
331 351 2,627 0.56 
341 351 2,608 0.46 
301 302 2632 0.59 
301 303 2598 0.58 
301 304 2615 0.48 
301 305 2615 0.47 
302 303 2587 0.58 
302 304 2603 0.59 
302 305 2601 0.56 
303 304 2573 0.50 
303 305 2579 0.46 
304 305 2590 0.49 


Mathematics 


Table I-2. 2013-14 NYSAA: Standards-Based Item Correlations for Grade 4 


Subject Ss ele N Correlation 
411 413 2888 0.53 
411 432 2954 0.45 
411 442 2880 0.47 


411 453 2951 0.48 


ae 413. -432~=S—s« 2866 0.33 
ig 413. 442~=—«-2798 0.47 


413 453 2861 0.48 
432 442 2860 0.40 
432 453 2925 0.42 
442 453-2854 0.44 
401 402 2,940 0.45 
401 403 2,914 0.59 
401 404 2,936 0.56 
401 405 2,927 0.45 
402 403 2,918 0.44 
402 404 2,936 0.45 
402 405 2,924 0.42 
403 404 2,916 0.55 
403 405 2,900 0.44 
404 405 2,927 0.44 
Science 411 422 2,894 0.49 


Mathematics 


2013-14 NYSAA Technical Report: Appendix I—Correlations Between Standards-Based Items 


Table I-3. 2013-14 NYSAA: Standards-Based Item Correlations for Grade 5 
Subject ee N Correlation 
512 523 2995 0.51 
512 533 2955 0.45 
512 5A1 2958 0.35 
512 552 2937 0.36 


ee 523 533 ©2974 0.50 
meee 523. 541 2975 0.40 


Mathematics 


Table I-4. 2013-14 NYSAA: Standards-Based Item Correlations for Grade 6 


; Pair of ; 
Subject Standards N Correlation 
611 621 3218 0.52 
611 631 3191 0.44 


611 641 3171 0.48 


ee 621 631 3169 0.50 
Rae 621 641 3141 0.49 


Mathematics 
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Table I-5. 2013-14 NYSAA: Standards-Based Item Correlations for Grade 7 
. Pair of , 
Subject Standanis N Correlation 


713 724 3200 0.47 


ae WA. “7B S14 0.47 
Rae? 724 741 3152 0.45 


Mathematics 


Table I-6. 2013-14 NYSAA: Standards-Based Item Correlations for Grade 8 
Subject a, N Correlation 
822 823 2969 0.51 
822 833 2981 0.54 
822 842 2927 0.46 
822 852 2981 0.48 


eee 823 833 3038 0.57 
ae 823 842 2983 0.49 


823 852 = =©3041 0.49 


Mathematics 


Science 813 832 2,985 0.48 
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Table I-7. 2013-14 NYSAA: Standards-Based Item Correlations for High School 


; Pair of : 
Subject Standards N Correlation 


911 921 2843 0.49 


English 921 931 2867 0.45 
Language Arts 921 942 2788 0.47 


Mathematics 


Science 921 931 2,912 0.58 
Social Studies 911 921 2,932 0.62 
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